[==========] Running 6 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 4 tests from MasterCertAuthorityTest
[ RUN      ] MasterCertAuthorityTest.CertAuthorityPersistsUponRestart
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 19:25:29.238864 21284 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.20.201.62:50411,127.20.201.61:35872,127.20.201.60:52076
I1014 19:25:29.249318 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417078us
I1014 19:25:29.250082 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:29.250155 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:29.250435 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:29.250545 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:29.253928 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root/instance:
uuid: "99714f99f26143dca7fd38e15b849c91"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.264638 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.010s	user 0.008s	sys 0.001s
I1014 19:25:29.265967 21284 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:29.266541 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:29.271674 21284 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I1014 19:25:29.271750 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
uuid: "99714f99f26143dca7fd38e15b849c91"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.271878 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:29.299134 21284 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536
I1014 19:25:29.354223 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:50411
I1014 19:25:29.354804 21284 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:29.354849 21284 webserver.cc:186] Document root disabled
I1014 19:25:29.355823 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:37234/
I1014 19:25:29.361562 21469 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:29.362426 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417078us
I1014 19:25:29.362707 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:29.362812 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:29.363072 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:29.363173 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:29.364094 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root/instance:
uuid: "e63b79beb7394619b5da13690787d902"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.377305 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.013s	user 0.003s	sys 0.007s
I1014 19:25:29.379253 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:29.385601 21284 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I1014 19:25:29.385679 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
uuid: "e63b79beb7394619b5da13690787d902"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.385766 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:29.383177 21469 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.422493 21469 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:29.423822 21469 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:35872: Network error: Client connection negotiation failed: client connection to 127.20.201.61:35872: connect: Connection refused (error 111)
I1014 19:25:29.467161 21469 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } attempt: 1
W1014 19:25:29.468562 21469 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:35872: Network error: Client connection negotiation failed: client connection to 127.20.201.61:35872: connect: Connection refused (error 111)
I1014 19:25:29.501981 21469 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } attempt: 2
W1014 19:25:29.503548 21469 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:35872: Network error: Client connection negotiation failed: client connection to 127.20.201.61:35872: connect: Connection refused (error 111)
I1014 19:25:29.519383 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:35872
I1014 19:25:29.519747 21284 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:29.519798 21284 webserver.cc:186] Document root disabled
I1014 19:25:29.520510 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:42247/
I1014 19:25:29.522686 21617 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:29.526114 21617 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.527532 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417078us
I1014 19:25:29.527887 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:29.528039 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:29.528264 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:29.528363 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:29.528937 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root/instance:
uuid: "efaa4987e3b34825a1ad301fd233e4bf"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.541080 21617 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.541290 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.012s	user 0.002s	sys 0.007s
I1014 19:25:29.545699 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:29.551851 21617 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.552500 21284 fs_manager.cc:419] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.003s
I1014 19:25:29.552652 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
uuid: "efaa4987e3b34825a1ad301fd233e4bf"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
W1014 19:25:29.553155 21617 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:52076: Network error: Client connection negotiation failed: client connection to 127.20.201.60:52076: connect: Connection refused (error 111)
I1014 19:25:29.553213 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:29.584393 21617 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } attempt: 1
W1014 19:25:29.585347 21617 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:52076: Network error: Client connection negotiation failed: client connection to 127.20.201.60:52076: connect: Connection refused (error 111)
I1014 19:25:29.613807 21469 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } attempt: 3
I1014 19:25:29.619560 21469 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:29.620517 21469 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:52076: Network error: Client connection negotiation failed: client connection to 127.20.201.60:52076: connect: Connection refused (error 111)
I1014 19:25:29.664652 21617 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } attempt: 2
W1014 19:25:29.665629 21617 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:52076: Network error: Client connection negotiation failed: client connection to 127.20.201.60:52076: connect: Connection refused (error 111)
I1014 19:25:29.665761 21469 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } attempt: 1
W1014 19:25:29.668898 21469 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:52076: Network error: Client connection negotiation failed: client connection to 127.20.201.60:52076: connect: Connection refused (error 111)
I1014 19:25:29.677774 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:52076
I1014 19:25:29.678067 21284 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:29.678107 21284 webserver.cc:186] Document root disabled
I1014 19:25:29.678807 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:59994/
I1014 19:25:29.679877 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0
I1014 19:25:29.680626 21693 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:29.682678 21693 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.689116 21693 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.694226 21693 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:29.703920 21693 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Bootstrap starting.
I1014 19:25:29.706693 21693 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: No blocks or log segments found. Creating new log.
I1014 19:25:29.707648 21693 log.cc:525] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Log is configured to *not* fsync() on all Append() calls
I1014 19:25:29.710265 21693 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: No bootstrap required, opened a new log
I1014 19:25:29.716528 21693 raft_consensus.cc:325] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.716774 21693 raft_consensus.cc:351] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:29.716858 21693 raft_consensus.cc:630] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaa4987e3b34825a1ad301fd233e4bf, State: Initialized, Role: FOLLOWER
I1014 19:25:29.717411 21693 consensus_queue.cc:228] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.719003 21697 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.719326 21697 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.718852 21693 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.719543 21693 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.720765 21693 sys_catalog.cc:424] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: configured and running, proceeding with master startup.
W1014 19:25:29.726299 21698 catalog_manager.cc:1084] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:29.726490 21698 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:29.744256 21469 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } attempt: 2
I1014 19:25:29.753878 21469 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Bootstrap starting.
I1014 19:25:29.756163 21469 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: No blocks or log segments found. Creating new log.
I1014 19:25:29.759649 21469 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: No bootstrap required, opened a new log
I1014 19:25:29.761117 21469 raft_consensus.cc:325] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.761502 21469 raft_consensus.cc:351] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:29.761639 21469 raft_consensus.cc:630] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99714f99f26143dca7fd38e15b849c91, State: Initialized, Role: FOLLOWER
I1014 19:25:29.762049 21469 consensus_queue.cc:228] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.763586 21469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.763988 21469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.764720 21469 sys_catalog.cc:424] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:29.767151 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1
I1014 19:25:29.767204 21701 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.768885 21701 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:25:29.770048 21703 catalog_manager.cc:1084] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:29.770151 21703 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:29.770911 21617 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } attempt: 3
I1014 19:25:29.780385 21617 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Bootstrap starting.
I1014 19:25:29.782655 21617 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: No blocks or log segments found. Creating new log.
I1014 19:25:29.785809 21617 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: No bootstrap required, opened a new log
I1014 19:25:29.787470 21617 raft_consensus.cc:325] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.787819 21617 raft_consensus.cc:351] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:29.787930 21617 raft_consensus.cc:630] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63b79beb7394619b5da13690787d902, State: Initialized, Role: FOLLOWER
I1014 19:25:29.788326 21617 consensus_queue.cc:228] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.789680 21617 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.790040 21617 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.790036 21707 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.790709 21707 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.791079 21617 sys_catalog.cc:424] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:29.797406 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2
W1014 19:25:29.799602 21709 catalog_manager.cc:1084] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:29.799738 21709 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W1014 19:25:29.802727 21415 tablet.cc:1720] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I1014 19:25:29.805616 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417078us
I1014 19:25:29.805887 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:29.806015 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:29.806181 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:29.806290 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:29.807140 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root/instance:
uuid: "bfb645a6e45043fe9b3933816423ce8e"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.815145 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.002s
I1014 19:25:29.816676 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:29.825536 21284 fs_manager.cc:419] Time spent opening block manager: real 0.006s	user 0.000s	sys 0.005s
I1014 19:25:29.825666 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
uuid: "bfb645a6e45043fe9b3933816423ce8e"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:29.825886 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:29.826794 21697 raft_consensus.cc:457] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:29.827121 21697 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.978s (starting election)
I1014 19:25:29.827309 21697 raft_consensus.cc:479] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.828526 21697 leader_election.cc:251] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 99714f99f26143dca7fd38e15b849c91 (127.20.201.62:50411)
I1014 19:25:29.829197 21697 leader_election.cc:251] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.830893 21701 raft_consensus.cc:457] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:29.831420 21701 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 2.165s (starting election)
I1014 19:25:29.830462 21569 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efaa4987e3b34825a1ad301fd233e4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902" is_pre_election: true
I1014 19:25:29.831682 21701 raft_consensus.cc:479] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.830890 21439 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efaa4987e3b34825a1ad301fd233e4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99714f99f26143dca7fd38e15b849c91" is_pre_election: true
I1014 19:25:29.832494 21439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 2.144s (vote granted)
I1014 19:25:29.832137 21701 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.832494 21569 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Snoozing failure detection for 1.855s (vote granted)
I1014 19:25:29.832818 21569 raft_consensus.cc:2255] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efaa4987e3b34825a1ad301fd233e4bf in term 0.
I1014 19:25:29.833098 21439 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efaa4987e3b34825a1ad301fd233e4bf in term 0.
I1014 19:25:29.833428 21701 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:29.834100 21638 leader_election.cc:387] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 pre-election: Vote granted by peer 99714f99f26143dca7fd38e15b849c91 (127.20.201.62:50411)
I1014 19:25:29.834245 21638 leader_election.cc:278] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I1014 19:25:29.834502 21641 leader_election.cc:387] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 pre-election: Vote granted by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.834498 21568 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902" is_pre_election: true
I1014 19:25:29.835369 21697 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.962s (election complete)
I1014 19:25:29.835670 21568 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Snoozing failure detection for 2.008s (vote granted)
I1014 19:25:29.835664 21697 raft_consensus.cc:2549] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I1014 19:25:29.835083 21670 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaa4987e3b34825a1ad301fd233e4bf" is_pre_election: true
I1014 19:25:29.836526 21697 raft_consensus.cc:457] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:29.836001 21568 raft_consensus.cc:2255] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99714f99f26143dca7fd38e15b849c91 in term 0.
I1014 19:25:29.836771 21697 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.790s (starting election)
I1014 19:25:29.836884 21697 raft_consensus.cc:2795] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:29.840417 21405 leader_election.cc:387] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 pre-election: Vote granted by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.840543 21405 leader_election.cc:278] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I1014 19:25:29.840756 21701 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 1.778s (election complete)
I1014 19:25:29.840883 21701 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I1014 19:25:29.840983 21701 raft_consensus.cc:457] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:29.841163 21701 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 2.108s (starting election)
I1014 19:25:29.841289 21701 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:29.841161 21697 raft_consensus.cc:479] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.841748 21670 raft_consensus.cc:2185] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 in current term 1: Already voted for candidate efaa4987e3b34825a1ad301fd233e4bf in this term.
I1014 19:25:29.843246 21697 leader_election.cc:251] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 election: Requesting vote from peer 99714f99f26143dca7fd38e15b849c91 (127.20.201.62:50411)
I1014 19:25:29.843910 21406 leader_election.cc:401] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 pre-election: Vote denied by peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076). Message: Invalid argument: T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 in current term 1: Already voted for candidate efaa4987e3b34825a1ad301fd233e4bf in this term.
I1014 19:25:29.845304 21439 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efaa4987e3b34825a1ad301fd233e4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99714f99f26143dca7fd38e15b849c91"
I1014 19:25:29.845518 21697 leader_election.cc:251] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 election: Requesting vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.846176 21568 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efaa4987e3b34825a1ad301fd233e4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902"
I1014 19:25:29.846549 21568 raft_consensus.cc:2795] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:29.846766 21568 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Snoozing failure detection for 1.565s (vote granted)
I1014 19:25:29.848317 21701 raft_consensus.cc:479] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.849009 21439 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate efaa4987e3b34825a1ad301fd233e4bf in current term 1: Already voted for candidate 99714f99f26143dca7fd38e15b849c91 in this term.
I1014 19:25:29.849027 21701 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 election: Requesting vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.851735 21569 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902"
I1014 19:25:29.852044 21638 leader_election.cc:401] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 election: Vote denied by peer 99714f99f26143dca7fd38e15b849c91 (127.20.201.62:50411). Message: Invalid argument: T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate efaa4987e3b34825a1ad301fd233e4bf in current term 1: Already voted for candidate 99714f99f26143dca7fd38e15b849c91 in this term.
I1014 19:25:29.852486 21701 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 election: Requesting vote from peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:29.853734 21670 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaa4987e3b34825a1ad301fd233e4bf"
I1014 19:25:29.854163 21670 raft_consensus.cc:2185] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 in current term 1: Already voted for candidate efaa4987e3b34825a1ad301fd233e4bf in this term.
I1014 19:25:29.855327 21406 leader_election.cc:401] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 election: Vote denied by peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076). Message: Invalid argument: T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 in current term 1: Already voted for candidate efaa4987e3b34825a1ad301fd233e4bf in this term.
I1014 19:25:29.855330 21568 raft_consensus.cc:2255] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efaa4987e3b34825a1ad301fd233e4bf in term 1.
I1014 19:25:29.855819 21569 raft_consensus.cc:2230] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 for term 1 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:29.855818 21641 leader_election.cc:387] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 election: Vote granted by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:29.856104 21641 leader_election.cc:278] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:29.856506 21697 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.630s (election complete)
I1014 19:25:29.856775 21697 raft_consensus.cc:2549] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader election won for term 1
I1014 19:25:29.857615 21405 leader_election.cc:401] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 election: Vote denied by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872). Message: Service unavailable: T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99714f99f26143dca7fd38e15b849c91 for term 1 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:29.857884 21405 leader_election.cc:278] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost.
I1014 19:25:29.858188 21701 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 1.534s (election complete)
I1014 19:25:29.858489 21701 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I1014 19:25:29.864114 21697 raft_consensus.cc:595] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 LEADER]: Becoming Leader. State: Replica: efaa4987e3b34825a1ad301fd233e4bf, State: Running, Role: LEADER
I1014 19:25:29.866699 21697 consensus_queue.cc:205] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:29.876442 21721 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader efaa4987e3b34825a1ad301fd233e4bf. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.877022 21721 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: LEADER
I1014 19:25:29.879858 21732 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:29.884387 21732 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:29.971192 21439 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Refusing update from remote peer efaa4987e3b34825a1ad301fd233e4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:29.971424 21284 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:29.972235 21569 raft_consensus.cc:1083] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Refusing update from remote peer efaa4987e3b34825a1ad301fd233e4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:29.972684 21721 consensus_queue.cc:963] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:29.972968 21697 consensus_queue.cc:963] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:29.984887 21707 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efaa4987e3b34825a1ad301fd233e4bf. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.985236 21707 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.986996 21701 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efaa4987e3b34825a1ad301fd233e4bf. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.988132 21701 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:29.992681 21761 mvcc.cc:203] Tried to move safe_time back from 6306080071549472768 to 6306080071141748736. Current Snapshot: MvccSnapshot[committed={T|T < 6306080071549472768}]
I1014 19:25:29.998821 21707 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:29.999264 21707 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.000625 21701 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.000973 21701 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.008424 21732 catalog_manager.cc:924] Generated new certificate authority record
I1014 19:25:30.008716 21721 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.009328 21697 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efaa4987e3b34825a1ad301fd233e4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.009891 21697 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: LEADER
I1014 19:25:30.010537 21732 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:30.011150 21721 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: LEADER
I1014 19:25:30.025720 21732 catalog_manager.cc:4062] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Generated new TSK 0
I1014 19:25:30.222190 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:34806
I1014 19:25:30.222537 21284 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:30.222594 21284 webserver.cc:186] Document root disabled
I1014 19:25:30.223194 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:56289/
I1014 19:25:30.257093 21906 heartbeater.cc:346] Connected to a master server at 127.20.201.62:50411
I1014 19:25:30.257310 21906 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.259352 21908 heartbeater.cc:346] Connected to a master server at 127.20.201.60:52076
I1014 19:25:30.259774 21908 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.259915 21906 heartbeater.cc:475] Master 127.20.201.62:50411 requested a full tablet report, sending...
I1014 19:25:30.260592 21908 heartbeater.cc:475] Master 127.20.201.60:52076 requested a full tablet report, sending...
I1014 19:25:30.262569 21660 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:34806)
I1014 19:25:30.265200 21427 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:34806)
I1014 19:25:30.265791 21907 heartbeater.cc:346] Connected to a master server at 127.20.201.61:35872
I1014 19:25:30.265903 21907 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.266932 21907 heartbeater.cc:475] Master 127.20.201.61:35872 requested a full tablet report, sending...
I1014 19:25:30.268193 21660 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.60:44443
I1014 19:25:30.268595 21547 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:34806)
I1014 19:25:30.269392 21284 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.034130035s
I1014 19:25:30.284114 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:30.302090 21284 ts_tablet_manager.cc:1118] Shutting down tablet manager...
I1014 19:25:30.308208 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:30.308323 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:30.308590 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:30.313427 21284 master.cc:249] Master@127.20.201.62:50411 shutting down...
I1014 19:25:30.327352 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:30.327618 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:30.327782 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: stopping tablet replica
I1014 19:25:30.328019 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:30.328164 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.328305 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.328435 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:30.328631 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:30.328721 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:30.328831 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:30.345512 21284 master.cc:260] Master@127.20.201.62:50411 shutdown complete.
I1014 19:25:30.350637 21284 master.cc:249] Master@127.20.201.61:35872 shutting down...
I1014 19:25:30.373034 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:30.373220 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:30.373363 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: stopping tablet replica
I1014 19:25:30.373512 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:30.373615 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.373715 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.373821 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:30.373931 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:30.374223 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:30.374311 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:30.396690 21284 master.cc:260] Master@127.20.201.61:35872 shutdown complete.
I1014 19:25:30.400323 21284 master.cc:249] Master@127.20.201.60:52076 shutting down...
I1014 19:25:30.413132 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 LEADER]: Raft consensus shutting down.
I1014 19:25:30.413265 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf -> Peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872): Closing peer: e63b79beb7394619b5da13690787d902
I1014 19:25:30.413399 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf -> Peer 99714f99f26143dca7fd38e15b849c91 (127.20.201.62:50411): Closing peer: 99714f99f26143dca7fd38e15b849c91
I1014 19:25:30.413784 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:30.414361 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: stopping tablet replica
I1014 19:25:30.414590 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:30.414698 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.414775 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:30.414837 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:30.414947 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:30.415021 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:30.415091 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:30.429983 21284 master.cc:260] Master@127.20.201.60:52076 shutdown complete.
I1014 19:25:30.438827 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417578us
I1014 19:25:30.439116 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:30.439225 21284 fs_manager.cc:266] Using existing metadata directory in first data directory
I1014 19:25:30.441589 21284 fs_manager.cc:399] Time spent opening directory manager: real 0.002s	user 0.001s	sys 0.001s
I1014 19:25:30.442543 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:30.445701 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:30.445797 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
uuid: "99714f99f26143dca7fd38e15b849c91"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:30.445937 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:30.574688 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:50411
I1014 19:25:30.574941 21284 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:30.574986 21284 webserver.cc:186] Document root disabled
I1014 19:25:30.575578 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:40918/
I1014 19:25:30.580116 22097 sys_catalog.cc:188] Verifying existing consensus state
I1014 19:25:30.580788 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417578us
I1014 19:25:30.581033 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:30.581182 21284 fs_manager.cc:266] Using existing metadata directory in first data directory
I1014 19:25:30.583787 21284 fs_manager.cc:399] Time spent opening directory manager: real 0.002s	user 0.001s	sys 0.000s
I1014 19:25:30.584302 22097 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Bootstrap starting.
I1014 19:25:30.584688 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:30.587864 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:30.587996 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
uuid: "e63b79beb7394619b5da13690787d902"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:30.588181 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:30.595011 22097 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=1} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I1014 19:25:30.596093 22097 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Bootstrap complete.
I1014 19:25:30.597404 22097 raft_consensus.cc:325] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.597762 22097 raft_consensus.cc:630] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99714f99f26143dca7fd38e15b849c91, State: Initialized, Role: FOLLOWER
I1014 19:25:30.598088 22097 consensus_queue.cc:228] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.598865 22097 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.599202 22097 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.599007 22105 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.599392 22105 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.599902 22097 sys_catalog.cc:424] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:30.605370 22106 catalog_manager.cc:525] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I1014 19:25:30.645601 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:35872
I1014 19:25:30.645911 21284 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:30.645961 21284 webserver.cc:186] Document root disabled
I1014 19:25:30.646621 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:34932/
I1014 19:25:30.650933 22155 sys_catalog.cc:188] Verifying existing consensus state
I1014 19:25:30.652737 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417578us
I1014 19:25:30.653013 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:30.653122 21284 fs_manager.cc:266] Using existing metadata directory in first data directory
I1014 19:25:30.653936 22155 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Bootstrap starting.
I1014 19:25:30.655226 21284 fs_manager.cc:399] Time spent opening directory manager: real 0.002s	user 0.001s	sys 0.000s
I1014 19:25:30.656419 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:30.660631 21284 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.000s
I1014 19:25:30.660845 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
uuid: "efaa4987e3b34825a1ad301fd233e4bf"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:30.661015 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:30.663933 22155 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=1} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I1014 19:25:30.664582 22155 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Bootstrap complete.
I1014 19:25:30.665552 22155 raft_consensus.cc:325] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.665814 22155 raft_consensus.cc:630] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63b79beb7394619b5da13690787d902, State: Initialized, Role: FOLLOWER
I1014 19:25:30.666142 22155 consensus_queue.cc:228] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.666998 22155 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.667446 22155 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.667173 22164 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.667737 22164 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.668295 22155 sys_catalog.cc:424] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:30.676501 22167 catalog_manager.cc:1082] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: acquiring CA information for follower catalog manager: success
I1014 19:25:30.678143 22167 catalog_manager.cc:1110] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:30.715592 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:52076
I1014 19:25:30.715912 21284 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:30.715957 21284 webserver.cc:186] Document root disabled
I1014 19:25:30.716555 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:35740/
I1014 19:25:30.716979 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0
I1014 19:25:30.717080 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1
I1014 19:25:30.717201 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2
I1014 19:25:30.719290 22223 sys_catalog.cc:188] Verifying existing consensus state
I1014 19:25:30.721102 22223 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Bootstrap starting.
I1014 19:25:30.727121 22223 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=1} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I1014 19:25:30.727614 22223 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Bootstrap complete.
I1014 19:25:30.728435 22223 raft_consensus.cc:325] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.728669 22223 raft_consensus.cc:630] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaa4987e3b34825a1ad301fd233e4bf, State: Initialized, Role: FOLLOWER
I1014 19:25:30.728902 22223 consensus_queue.cc:228] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:30.729537 22223 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.729867 22223 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.729748 22239 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:30.730406 22223 sys_catalog.cc:424] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:30.730584 22239 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:30.735527 22240 catalog_manager.cc:525] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I1014 19:25:30.740723 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 417578us
I1014 19:25:30.740885 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:30.740933 21284 fs_manager.cc:266] Using existing metadata directory in first data directory
I1014 19:25:30.742573 21284 fs_manager.cc:399] Time spent opening directory manager: real 0.001s	user 0.002s	sys 0.000s
I1014 19:25:30.743052 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:30.745668 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I1014 19:25:30.745733 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
uuid: "bfb645a6e45043fe9b3933816423ce8e"
format_stamp: "Formatted at 2018-10-15 02:25:29 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:30.745832 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityPersistsUponRestart.1539570329182132-21284/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:30.810812 21284 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:30.928782 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:47731
I1014 19:25:30.929045 21284 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:30.929093 21284 webserver.cc:186] Document root disabled
I1014 19:25:30.929795 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:33634/
I1014 19:25:30.963037 22404 heartbeater.cc:346] Connected to a master server at 127.20.201.62:50411
I1014 19:25:30.963182 22404 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.963737 22404 heartbeater.cc:475] Master 127.20.201.62:50411 requested a full tablet report, sending...
I1014 19:25:30.965183 22064 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:47731)
I1014 19:25:30.970129 22405 heartbeater.cc:346] Connected to a master server at 127.20.201.61:35872
I1014 19:25:30.970279 22405 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.970754 22405 heartbeater.cc:475] Master 127.20.201.61:35872 requested a full tablet report, sending...
I1014 19:25:30.972357 22122 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:47731)
I1014 19:25:30.972667 22408 heartbeater.cc:346] Connected to a master server at 127.20.201.60:52076
I1014 19:25:30.972795 22408 heartbeater.cc:426] Registering TS with master...
I1014 19:25:30.973068 22408 heartbeater.cc:475] Master 127.20.201.60:52076 requested a full tablet report, sending...
I1014 19:25:30.973923 22186 ts_manager.cc:98] Registered new tserver with Master: bfb645a6e45043fe9b3933816423ce8e (127.20.201.1:47731)
I1014 19:25:30.974107 21284 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.01439673s
I1014 19:25:31.608305 22106 catalog_manager.cc:1082] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: acquiring CA information for follower catalog manager: success
I1014 19:25:31.609254 22106 catalog_manager.cc:1110] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:31.738492 22240 catalog_manager.cc:1082] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: acquiring CA information for follower catalog manager: success
I1014 19:25:31.739434 22240 catalog_manager.cc:1110] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:32.327217 23013 raft_consensus.cc:457] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:32.327319 23013 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 1.836s (starting election)
I1014 19:25:32.327455 23013 raft_consensus.cc:479] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:32.328044 23013 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:32.329362 23013 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:32.342563 22132 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902" is_pre_election: true
I1014 19:25:32.342767 22132 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Snoozing failure detection for 1.913s (vote granted)
I1014 19:25:32.342846 22132 raft_consensus.cc:2255] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99714f99f26143dca7fd38e15b849c91 in term 1.
I1014 19:25:32.343250 22018 leader_election.cc:387] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 pre-election: Vote granted by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:32.343360 22018 leader_election.cc:278] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I1014 19:25:32.343603 23013 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 1.542s (election complete)
I1014 19:25:32.343715 23013 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I1014 19:25:32.343821 23013 raft_consensus.cc:457] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:32.343917 23013 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 2.005s (starting election)
I1014 19:25:32.344063 23013 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:32.345178 22200 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "efaa4987e3b34825a1ad301fd233e4bf" is_pre_election: true
I1014 19:25:32.345561 22200 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.817s (vote granted)
I1014 19:25:32.345705 22200 raft_consensus.cc:2255] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99714f99f26143dca7fd38e15b849c91 in term 1.
I1014 19:25:32.346242 22021 leader_election.cc:387] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 pre-election: Vote granted by peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:32.346839 23013 raft_consensus.cc:479] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:32.347177 23013 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 election: Requesting vote from peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:32.347524 23013 leader_election.cc:251] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 election: Requesting vote from peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:32.347719 22132 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e63b79beb7394619b5da13690787d902"
I1014 19:25:32.347862 22132 raft_consensus.cc:2795] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:32.347985 22132 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: Snoozing failure detection for 1.677s (vote granted)
I1014 19:25:32.348124 22200 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99714f99f26143dca7fd38e15b849c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "efaa4987e3b34825a1ad301fd233e4bf"
I1014 19:25:32.348263 22200 raft_consensus.cc:2795] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:32.348392 22200 raft_consensus.cc:2743] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: Snoozing failure detection for 1.727s (vote granted)
I1014 19:25:32.350419 22200 raft_consensus.cc:2255] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99714f99f26143dca7fd38e15b849c91 in term 2.
I1014 19:25:32.350716 22021 leader_election.cc:387] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 election: Vote granted by peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076)
I1014 19:25:32.350819 22021 leader_election.cc:278] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I1014 19:25:32.350894 22132 raft_consensus.cc:2255] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99714f99f26143dca7fd38e15b849c91 in term 2.
I1014 19:25:32.351053 23013 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Snoozing failure detection for 2.143s (election complete)
I1014 19:25:32.351249 23013 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 2 FOLLOWER]: Leader election won for term 2
I1014 19:25:32.351250 22018 leader_election.cc:387] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [CANDIDATE]: Term 2 election: Vote granted by peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872)
I1014 19:25:32.351729 23013 raft_consensus.cc:595] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 2 LEADER]: Becoming Leader. State: Replica: 99714f99f26143dca7fd38e15b849c91, State: Running, Role: LEADER
I1014 19:25:32.352053 23013 consensus_queue.cc:205] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } }
I1014 19:25:32.353667 23029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99714f99f26143dca7fd38e15b849c91. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.354054 23029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:32.354998 23033 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:32.357187 23033 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:32.358649 23033 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:32.359427 23033 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: Loaded TSK: 0
I1014 19:25:32.366968 22200 raft_consensus.cc:1083] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 2 FOLLOWER]: Refusing update from remote peer 99714f99f26143dca7fd38e15b849c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I1014 19:25:32.366968 22132 raft_consensus.cc:1083] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 2 FOLLOWER]: Refusing update from remote peer 99714f99f26143dca7fd38e15b849c91: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I1014 19:25:32.367631 23013 consensus_queue.cc:963] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I1014 19:25:32.367854 23029 consensus_queue.cc:963] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I1014 19:25:32.391813 23043 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99714f99f26143dca7fd38e15b849c91. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.396876 23043 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.396445 23042 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99714f99f26143dca7fd38e15b849c91. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.397188 23042 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.400605 23013 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.410164 23013 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:32.410537 23013 sys_catalog.cc:337] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.410818 23013 sys_catalog.cc:340] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:32.411870 23043 sys_catalog.cc:337] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.412179 23043 sys_catalog.cc:340] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.413187 23042 sys_catalog.cc:337] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "99714f99f26143dca7fd38e15b849c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99714f99f26143dca7fd38e15b849c91" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50411 } } peers { permanent_uuid: "e63b79beb7394619b5da13690787d902" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 35872 } } peers { permanent_uuid: "efaa4987e3b34825a1ad301fd233e4bf" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 52076 } } }
I1014 19:25:32.413473 23042 sys_catalog.cc:340] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.499189 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:32.511142 21284 ts_tablet_manager.cc:1118] Shutting down tablet manager...
I1014 19:25:32.523957 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:32.524010 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:32.524154 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:32.526417 21284 master.cc:249] Master@127.20.201.62:50411 shutting down...
I1014 19:25:32.535465 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 2 LEADER]: Raft consensus shutting down.
I1014 19:25:32.535562 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 -> Peer efaa4987e3b34825a1ad301fd233e4bf (127.20.201.60:52076): Closing peer: efaa4987e3b34825a1ad301fd233e4bf
I1014 19:25:32.535641 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 -> Peer e63b79beb7394619b5da13690787d902 (127.20.201.61:35872): Closing peer: e63b79beb7394619b5da13690787d902
I1014 19:25:32.535846 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91 [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:32.536144 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P 99714f99f26143dca7fd38e15b849c91: stopping tablet replica
I1014 19:25:32.536367 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:32.536442 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.536532 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.536589 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:32.536662 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:32.536712 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:32.536762 21284 maintenance_manager.cc:237] P 99714f99f26143dca7fd38e15b849c91: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:32.542068 21284 master.cc:260] Master@127.20.201.62:50411 shutdown complete.
I1014 19:25:32.544497 21284 master.cc:249] Master@127.20.201.61:35872 shutting down...
I1014 19:25:32.553889 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 2 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:32.554016 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902 [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:32.554101 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P e63b79beb7394619b5da13690787d902: stopping tablet replica
I1014 19:25:32.554183 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:32.554246 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.554329 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.554390 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:32.554460 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:32.554534 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:32.554584 21284 maintenance_manager.cc:237] P e63b79beb7394619b5da13690787d902: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:32.558558 21284 master.cc:260] Master@127.20.201.61:35872 shutdown complete.
I1014 19:25:32.560734 21284 master.cc:249] Master@127.20.201.60:52076 shutting down...
I1014 19:25:32.570695 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 2 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:32.570813 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:32.570895 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P efaa4987e3b34825a1ad301fd233e4bf: stopping tablet replica
I1014 19:25:32.570978 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:32.571038 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.571097 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:32.571149 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:32.571213 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:32.571254 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:32.571290 21284 maintenance_manager.cc:237] P efaa4987e3b34825a1ad301fd233e4bf: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:32.584367 21284 master.cc:260] Master@127.20.201.60:52076 shutdown complete.
[       OK ] MasterCertAuthorityTest.CertAuthorityPersistsUponRestart (3357 ms)
[ RUN      ] MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch
I1014 19:25:32.590950 21284 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.20.201.62:46699,127.20.201.61:51869,127.20.201.60:60670
I1014 19:25:32.594064 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 418578us
I1014 19:25:32.594220 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:32.594267 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:32.594390 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:32.594429 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:32.594887 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-0-root/instance:
uuid: "a28067b582134222b182c2105d56d160"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.598364 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.002s
I1014 19:25:32.599093 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:32.600929 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I1014 19:25:32.600993 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-0-root
uuid: "a28067b582134222b182c2105d56d160"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.601076 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:32.671783 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:46699
I1014 19:25:32.672108 21284 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:32.672161 21284 webserver.cc:186] Document root disabled
I1014 19:25:32.672682 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:59004/
I1014 19:25:32.674028 23167 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:32.676375 23167 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.678345 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 418578us
I1014 19:25:32.678716 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:32.678843 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:32.679077 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:32.679203 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:32.679986 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-1-root/instance:
uuid: "327d629e926a4ee3abb197c0677f8b77"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.687532 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.007s	user 0.003s	sys 0.002s
I1014 19:25:32.689036 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:32.689234 23167 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:32.691395 23167 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:51869: Network error: Client connection negotiation failed: client connection to 127.20.201.61:51869: connect: Connection refused (error 111)
I1014 19:25:32.691771 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:32.691877 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-1-root
uuid: "327d629e926a4ee3abb197c0677f8b77"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.692030 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-1-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-1-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:32.714541 23167 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } attempt: 1
W1014 19:25:32.715584 23167 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:51869: Network error: Client connection negotiation failed: client connection to 127.20.201.61:51869: connect: Connection refused (error 111)
I1014 19:25:32.753839 23167 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } attempt: 2
W1014 19:25:32.754839 23167 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:51869: Network error: Client connection negotiation failed: client connection to 127.20.201.61:51869: connect: Connection refused (error 111)
I1014 19:25:32.796507 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:51869
I1014 19:25:32.796813 21284 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:32.796851 21284 webserver.cc:186] Document root disabled
I1014 19:25:32.797435 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:57972/
I1014 19:25:32.798854 23288 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:32.802453 23288 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.802647 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 418578us
I1014 19:25:32.802878 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:32.802969 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:32.803162 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:32.803253 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:32.804095 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-2-root/instance:
uuid: "20cbaaf9e32c4a18b8a721b9092c9a05"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.809779 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
I1014 19:25:32.810755 23288 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.811069 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:32.815032 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:32.815212 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-2-root
uuid: "20cbaaf9e32c4a18b8a721b9092c9a05"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.815439 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-2-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-2-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:32.820991 23288 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:32.821923 23288 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:60670: Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:32.851163 23288 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } attempt: 1
W1014 19:25:32.852032 23288 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:60670: Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:32.857100 23167 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } attempt: 3
I1014 19:25:32.862504 23167 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:32.863556 23167 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:60670: Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:32.890064 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:60670
I1014 19:25:32.890332 21284 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:32.890379 21284 webserver.cc:186] Document root disabled
I1014 19:25:32.890882 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:35132/
I1014 19:25:32.891330 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0
I1014 19:25:32.892927 23347 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:32.895576 23347 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.897269 23288 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } attempt: 2
I1014 19:25:32.902122 23347 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.904819 23167 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } attempt: 1
I1014 19:25:32.908854 23347 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:32.912700 23288 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Bootstrap starting.
I1014 19:25:32.915382 23288 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: No blocks or log segments found. Creating new log.
I1014 19:25:32.917886 23167 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Bootstrap starting.
I1014 19:25:32.920521 23167 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: No blocks or log segments found. Creating new log.
I1014 19:25:32.921759 23288 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: No bootstrap required, opened a new log
I1014 19:25:32.923104 23288 raft_consensus.cc:325] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.923360 23288 raft_consensus.cc:351] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:32.923496 23288 raft_consensus.cc:630] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327d629e926a4ee3abb197c0677f8b77, State: Initialized, Role: FOLLOWER
I1014 19:25:32.923776 23288 consensus_queue.cc:228] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.923969 23347 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Bootstrap starting.
I1014 19:25:32.925926 23347 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: No blocks or log segments found. Creating new log.
I1014 19:25:32.929844 23288 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.934643 23347 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: No bootstrap required, opened a new log
I1014 19:25:32.934649 23288 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.934309 23355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.935195 23355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.935451 23288 sys_catalog.cc:424] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:32.935950 23347 raft_consensus.cc:325] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.936190 23347 raft_consensus.cc:351] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:32.936282 23347 raft_consensus.cc:630] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cbaaf9e32c4a18b8a721b9092c9a05, State: Initialized, Role: FOLLOWER
I1014 19:25:32.936534 23347 consensus_queue.cc:228] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.937033 23167 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: No bootstrap required, opened a new log
I1014 19:25:32.937963 23167 raft_consensus.cc:325] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.938208 23167 raft_consensus.cc:351] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:32.938371 23167 raft_consensus.cc:630] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28067b582134222b182c2105d56d160, State: Initialized, Role: FOLLOWER
I1014 19:25:32.938912 23167 consensus_queue.cc:228] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:32.939803 23347 sys_catalog.cc:337] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.940397 23347 sys_catalog.cc:340] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.941098 23347 sys_catalog.cc:424] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: configured and running, proceeding with master startup.
W1014 19:25:32.941553 23357 catalog_manager.cc:1084] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:32.941648 23357 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:32.942080 23167 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.942595 23167 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.940141 23358 sys_catalog.cc:337] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.942713 23358 sys_catalog.cc:340] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.942807 23359 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:32.943451 23359 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:32.943455 23167 sys_catalog.cc:424] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:32.945917 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1
I1014 19:25:32.946113 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2
W1014 19:25:32.948422 23361 catalog_manager.cc:1084] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:32.948568 23361 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W1014 19:25:32.949720 23360 catalog_manager.cc:1084] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:32.949815 23360 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:32.951989 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 418578us
I1014 19:25:32.952167 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:32.952217 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:32.952324 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:32.952364 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:32.952874 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/ts-0-root/instance:
uuid: "5a663eeb01394bf08607ab93b2e5aa7f"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.956957 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I1014 19:25:32.957695 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:32.960137 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:32.960209 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/ts-0-root
uuid: "5a663eeb01394bf08607ab93b2e5aa7f"
format_stamp: "Formatted at 2018-10-15 02:25:32 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:32.960289 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/ts-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/ts-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch.1539570329182132-21284/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:32.990427 21284 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:33.027015 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:55999
I1014 19:25:33.027318 21284 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:33.027351 21284 webserver.cc:186] Document root disabled
I1014 19:25:33.027997 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:43962/
I1014 19:25:33.053439 23428 heartbeater.cc:346] Connected to a master server at 127.20.201.62:46699
I1014 19:25:33.053611 23428 heartbeater.cc:426] Registering TS with master...
I1014 19:25:33.054244 23428 heartbeater.cc:475] Master 127.20.201.62:46699 requested a full tablet report, sending...
I1014 19:25:33.055477 23429 heartbeater.cc:346] Connected to a master server at 127.20.201.61:51869
I1014 19:25:33.055640 23429 heartbeater.cc:426] Registering TS with master...
I1014 19:25:33.056429 23429 heartbeater.cc:475] Master 127.20.201.61:51869 requested a full tablet report, sending...
I1014 19:25:33.056435 23114 ts_manager.cc:98] Registered new tserver with Master: 5a663eeb01394bf08607ab93b2e5aa7f (127.20.201.1:55999)
I1014 19:25:33.057665 23430 heartbeater.cc:346] Connected to a master server at 127.20.201.60:60670
I1014 19:25:33.057793 23430 heartbeater.cc:426] Registering TS with master...
I1014 19:25:33.057895 23249 ts_manager.cc:98] Registered new tserver with Master: 5a663eeb01394bf08607ab93b2e5aa7f (127.20.201.1:55999)
I1014 19:25:33.058254 23430 heartbeater.cc:475] Master 127.20.201.60:60670 requested a full tablet report, sending...
I1014 19:25:33.059226 23314 ts_manager.cc:98] Registered new tserver with Master: 5a663eeb01394bf08607ab93b2e5aa7f (127.20.201.1:55999)
I1014 19:25:33.059953 21284 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.009748159s
I1014 19:25:33.135684 23358 raft_consensus.cc:457] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:33.135799 23358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Snoozing failure detection for 1.677s (starting election)
I1014 19:25:33.135917 23358 raft_consensus.cc:479] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:33.136463 23358 leader_election.cc:251] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699)
I1014 19:25:33.136760 23358 leader_election.cc:251] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:33.137569 23264 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327d629e926a4ee3abb197c0677f8b77" is_pre_election: true
I1014 19:25:33.137467 23129 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28067b582134222b182c2105d56d160" is_pre_election: true
I1014 19:25:33.137989 23264 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 2.152s (vote granted)
I1014 19:25:33.137989 23129 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 1.982s (vote granted)
I1014 19:25:33.138214 23129 raft_consensus.cc:2255] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cbaaf9e32c4a18b8a721b9092c9a05 in term 0.
I1014 19:25:33.138211 23264 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cbaaf9e32c4a18b8a721b9092c9a05 in term 0.
I1014 19:25:33.138718 23293 leader_election.cc:387] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699)
I1014 19:25:33.138818 23293 leader_election.cc:278] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I1014 19:25:33.138949 23295 leader_election.cc:387] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:33.139068 23358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Snoozing failure detection for 1.962s (election complete)
I1014 19:25:33.139145 23358 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I1014 19:25:33.139199 23358 raft_consensus.cc:457] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:33.139263 23358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Snoozing failure detection for 2.106s (starting election)
I1014 19:25:33.139322 23358 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:33.140888 23358 raft_consensus.cc:479] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:33.141300 23358 leader_election.cc:251] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 election: Requesting vote from peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699)
I1014 19:25:33.141520 23358 leader_election.cc:251] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 election: Requesting vote from peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:33.141890 23129 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28067b582134222b182c2105d56d160"
I1014 19:25:33.142067 23129 raft_consensus.cc:2795] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:33.142205 23129 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 2.175s (vote granted)
I1014 19:25:33.142263 23264 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327d629e926a4ee3abb197c0677f8b77"
I1014 19:25:33.142457 23264 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:33.142601 23264 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 1.976s (vote granted)
I1014 19:25:33.145277 23264 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cbaaf9e32c4a18b8a721b9092c9a05 in term 1.
I1014 19:25:33.145277 23129 raft_consensus.cc:2255] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cbaaf9e32c4a18b8a721b9092c9a05 in term 1.
I1014 19:25:33.145666 23293 leader_election.cc:387] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 election: Vote granted by peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699)
I1014 19:25:33.145761 23293 leader_election.cc:278] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:33.145876 23295 leader_election.cc:387] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [CANDIDATE]: Term 1 election: Vote granted by peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:33.145984 23358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Snoozing failure detection for 2.069s (election complete)
I1014 19:25:33.146088 23358 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 1 FOLLOWER]: Leader election won for term 1
I1014 19:25:33.146431 23358 raft_consensus.cc:595] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 1 LEADER]: Becoming Leader. State: Replica: 20cbaaf9e32c4a18b8a721b9092c9a05, State: Running, Role: LEADER
I1014 19:25:33.146793 23358 consensus_queue.cc:205] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:33.148063 23444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cbaaf9e32c4a18b8a721b9092c9a05. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.148628 23444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:33.149798 23446 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:33.151615 23446 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:33.173307 23264 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Refusing update from remote peer 20cbaaf9e32c4a18b8a721b9092c9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:33.173401 23129 raft_consensus.cc:1083] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Refusing update from remote peer 20cbaaf9e32c4a18b8a721b9092c9a05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:33.173926 23358 consensus_queue.cc:963] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:33.174291 23444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:33.185029 23355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cbaaf9e32c4a18b8a721b9092c9a05. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.185686 23355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:33.186790 23359 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20cbaaf9e32c4a18b8a721b9092c9a05. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.187309 23359 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:33.198022 23355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.198539 23355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:33.200704 23359 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.201002 23359 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:33.197268 23358 sys_catalog.cc:337] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.202642 23358 sys_catalog.cc:340] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:33.202847 23358 sys_catalog.cc:337] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:33.203068 23358 sys_catalog.cc:340] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:33.204242 23446 catalog_manager.cc:924] Generated new certificate authority record
I1014 19:25:33.206377 23446 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:33.218165 23446 catalog_manager.cc:4062] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: Generated new TSK 0
I1014 19:25:33.265785 21284 master.cc:249] Master@127.20.201.60:60670 shutting down...
I1014 19:25:33.274389 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 1 LEADER]: Raft consensus shutting down.
I1014 19:25:33.274477 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 -> Peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869): Closing peer: 327d629e926a4ee3abb197c0677f8b77
I1014 19:25:33.274597 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 -> Peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699): Closing peer: a28067b582134222b182c2105d56d160
I1014 19:25:33.274844 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:33.275136 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P 20cbaaf9e32c4a18b8a721b9092c9a05: stopping tablet replica
I1014 19:25:33.275324 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:33.275396 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:33.275465 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:33.275557 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:33.275633 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:33.275682 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:33.275739 21284 maintenance_manager.cc:237] P 20cbaaf9e32c4a18b8a721b9092c9a05: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:33.281949 21284 master.cc:260] Master@127.20.201.60:60670 shutdown complete.
I1014 19:25:33.945644 23357 catalog_manager.cc:1082] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: acquiring CA information for follower catalog manager: success
I1014 19:25:33.946916 23357 catalog_manager.cc:1110] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:33.951912 23361 catalog_manager.cc:1082] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: acquiring CA information for follower catalog manager: success
I1014 19:25:33.953088 23361 catalog_manager.cc:1110] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W1014 19:25:34.060904 23430 heartbeater.cc:587] Failed to heartbeat to 127.20.201.60:60670: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:34.723248 23677 raft_consensus.cc:457] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20cbaaf9e32c4a18b8a721b9092c9a05)
I1014 19:25:34.723394 23677 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 2.076s (starting election)
I1014 19:25:34.723557 23677 raft_consensus.cc:479] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:34.724140 23677 leader_election.cc:251] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:34.724481 23677 leader_election.cc:251] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670)
I1014 19:25:34.724934 23678 raft_consensus.cc:457] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20cbaaf9e32c4a18b8a721b9092c9a05)
I1014 19:25:34.725046 23678 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 1.524s (starting election)
I1014 19:25:34.725589 23264 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a28067b582134222b182c2105d56d160" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "327d629e926a4ee3abb197c0677f8b77" is_pre_election: true
I1014 19:25:34.725428 23678 raft_consensus.cc:479] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:34.726089 23264 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 2.129s (vote granted)
I1014 19:25:34.726240 23678 leader_election.cc:251] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699)
I1014 19:25:34.726240 23264 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a28067b582134222b182c2105d56d160 in term 1.
W1014 19:25:34.726613 23068 leader_election.cc:307] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670): Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:34.726835 23067 leader_election.cc:387] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:34.726904 23067 leader_election.cc:278] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I1014 19:25:34.727138 23677 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 1.653s (election complete)
I1014 19:25:34.727267 23677 raft_consensus.cc:2549] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Leader pre-election won for term 2
I1014 19:25:34.727268 23678 leader_election.cc:251] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670)
I1014 19:25:34.727459 23677 raft_consensus.cc:457] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 20cbaaf9e32c4a18b8a721b9092c9a05)
I1014 19:25:34.727457 23129 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "327d629e926a4ee3abb197c0677f8b77" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a28067b582134222b182c2105d56d160" is_pre_election: true
I1014 19:25:34.727591 23677 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 1.854s (starting election)
I1014 19:25:34.727686 23677 raft_consensus.cc:2795] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 1 FOLLOWER]: Advancing to term 2
W1014 19:25:34.728446 23183 leader_election.cc:307] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670): Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:34.730464 23677 raft_consensus.cc:479] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:34.730885 23129 raft_consensus.cc:2185] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 327d629e926a4ee3abb197c0677f8b77 in current term 2: Already voted for candidate a28067b582134222b182c2105d56d160 in this term.
I1014 19:25:34.731143 23677 leader_election.cc:251] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 election: Requesting vote from peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:34.731461 23677 leader_election.cc:251] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 election: Requesting vote from peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670)
I1014 19:25:34.731461 23180 leader_election.cc:401] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a28067b582134222b182c2105d56d160 (127.20.201.62:46699). Message: Invalid argument: T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 327d629e926a4ee3abb197c0677f8b77 in current term 2: Already voted for candidate a28067b582134222b182c2105d56d160 in this term.
I1014 19:25:34.731591 23180 leader_election.cc:278] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost.
I1014 19:25:34.731936 23678 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 2.158s (election complete)
I1014 19:25:34.732071 23678 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 1 FOLLOWER]: Advancing to term 2
W1014 19:25:34.732235 23068 leader_election.cc:307] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670): Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111)
I1014 19:25:34.732228 23264 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a28067b582134222b182c2105d56d160" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "327d629e926a4ee3abb197c0677f8b77"
I1014 19:25:34.733677 23678 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I1014 19:25:34.733824 23264 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: Snoozing failure detection for 3.012s (vote granted)
I1014 19:25:34.735076 23264 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a28067b582134222b182c2105d56d160 in term 2.
I1014 19:25:34.735440 23067 leader_election.cc:387] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 election: Vote granted by peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869)
I1014 19:25:34.735500 23067 leader_election.cc:278] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I1014 19:25:34.735694 23677 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Snoozing failure detection for 1.851s (election complete)
I1014 19:25:34.735791 23677 raft_consensus.cc:2549] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 FOLLOWER]: Leader election won for term 2
I1014 19:25:34.736205 23677 raft_consensus.cc:595] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 LEADER]: Becoming Leader. State: Replica: a28067b582134222b182c2105d56d160, State: Running, Role: LEADER
I1014 19:25:34.736541 23677 consensus_queue.cc:205] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } }
I1014 19:25:34.738106 23681 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28067b582134222b182c2105d56d160. Latest consensus state: current_term: 2 leader_uuid: "a28067b582134222b182c2105d56d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:34.738518 23681 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:34.739102 23683 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:34.740777 23683 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:34.741834 23683 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:34.742575 23683 catalog_manager.cc:4073] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: Loaded TSK: 0
I1014 19:25:34.745005 23264 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 2 FOLLOWER]: Refusing update from remote peer a28067b582134222b182c2105d56d160: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W1014 19:25:34.745004 23068 consensus_peers.cc:433] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 -> Peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670): Couldn't send request to peer 20cbaaf9e32c4a18b8a721b9092c9a05 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.20.201.60:60670: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I1014 19:25:34.745563 23681 consensus_queue.cc:963] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I1014 19:25:34.747694 23678 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28067b582134222b182c2105d56d160. Latest consensus state: current_term: 2 leader_uuid: "a28067b582134222b182c2105d56d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:34.747973 23678 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:34.749874 23677 sys_catalog.cc:337] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a28067b582134222b182c2105d56d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:34.750371 23677 sys_catalog.cc:340] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:34.750847 23678 sys_catalog.cc:337] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a28067b582134222b182c2105d56d160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28067b582134222b182c2105d56d160" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 46699 } } peers { permanent_uuid: "327d629e926a4ee3abb197c0677f8b77" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 51869 } } peers { permanent_uuid: "20cbaaf9e32c4a18b8a721b9092c9a05" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 60670 } } }
I1014 19:25:34.751106 23678 sys_catalog.cc:340] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:34.795146 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:34.806084 21284 ts_tablet_manager.cc:1118] Shutting down tablet manager...
I1014 19:25:34.818747 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:34.818843 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:34.819167 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:34.821665 21284 master.cc:249] Master@127.20.201.62:46699 shutting down...
I1014 19:25:34.831254 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 LEADER]: Raft consensus shutting down.
I1014 19:25:34.831360 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 -> Peer 20cbaaf9e32c4a18b8a721b9092c9a05 (127.20.201.60:60670): Closing peer: 20cbaaf9e32c4a18b8a721b9092c9a05
I1014 19:25:34.831471 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 -> Peer 327d629e926a4ee3abb197c0677f8b77 (127.20.201.61:51869): Closing peer: 327d629e926a4ee3abb197c0677f8b77
I1014 19:25:34.831670 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160 [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:34.831878 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P a28067b582134222b182c2105d56d160: stopping tablet replica
I1014 19:25:34.832074 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:34.832150 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:34.832207 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:34.832278 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:34.832345 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:34.832401 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:34.832469 21284 maintenance_manager.cc:237] P a28067b582134222b182c2105d56d160: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:34.836938 21284 master.cc:260] Master@127.20.201.62:46699 shutdown complete.
I1014 19:25:34.839432 21284 master.cc:249] Master@127.20.201.61:51869 shutting down...
I1014 19:25:34.848595 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 2 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:34.848707 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77 [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:34.848840 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P 327d629e926a4ee3abb197c0677f8b77: stopping tablet replica
I1014 19:25:34.848917 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:34.848985 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:34.849040 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:34.849104 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:34.849164 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:34.849218 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:34.849267 21284 maintenance_manager.cc:237] P 327d629e926a4ee3abb197c0677f8b77: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:34.862857 21284 master.cc:260] Master@127.20.201.61:51869 shutdown complete.
[       OK ] MasterCertAuthorityTest.CertAuthorityOnLeaderRoleSwitch (2279 ms)
[ RUN      ] MasterCertAuthorityTest.RefuseToSignInvalidCSR
I1014 19:25:34.869273 21284 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.20.201.62:50565,127.20.201.61:44802,127.20.201.60:55302
I1014 19:25:34.872360 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 419578us
I1014 19:25:34.872555 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:34.872614 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:34.872740 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:34.872792 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:34.873234 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-0-root/instance:
uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe"
format_stamp: "Formatted at 2018-10-15 02:25:34 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:34.876785 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I1014 19:25:34.877461 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:34.879349 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
I1014 19:25:34.879408 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-0-root
uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe"
format_stamp: "Formatted at 2018-10-15 02:25:34 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:34.879549 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:34.940094 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:50565
I1014 19:25:34.940485 21284 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:34.940549 21284 webserver.cc:186] Document root disabled
I1014 19:25:34.941694 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:57554/
I1014 19:25:34.942899 23790 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:34.945837 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 419578us
I1014 19:25:34.945806 23790 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:34.946064 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:34.946142 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:34.946363 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:34.946481 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:34.947356 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-1-root/instance:
uuid: "ec3fc2ca813e4055b02ec0351fa5a70c"
format_stamp: "Formatted at 2018-10-15 02:25:34 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:34.953717 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.006s	user 0.000s	sys 0.004s
I1014 19:25:34.955633 23790 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:34.955708 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
W1014 19:25:34.956779 23790 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:44802: Network error: Client connection negotiation failed: client connection to 127.20.201.61:44802: connect: Connection refused (error 111)
I1014 19:25:34.958093 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I1014 19:25:34.958189 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-1-root
uuid: "ec3fc2ca813e4055b02ec0351fa5a70c"
format_stamp: "Formatted at 2018-10-15 02:25:34 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:34.958323 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-1-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-1-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:34.985026 23790 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } attempt: 1
W1014 19:25:34.986089 23790 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:44802: Network error: Client connection negotiation failed: client connection to 127.20.201.61:44802: connect: Connection refused (error 111)
I1014 19:25:35.027333 23790 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } attempt: 2
W1014 19:25:35.028473 23790 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:44802: Network error: Client connection negotiation failed: client connection to 127.20.201.61:44802: connect: Connection refused (error 111)
I1014 19:25:35.056831 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:44802
I1014 19:25:35.057150 21284 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:35.057526 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.058087 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:52069/
I1014 19:25:35.059586 23922 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:35.062328 23922 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.064190 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
I1014 19:25:35.064555 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.064724 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.064961 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.065074 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.065868 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-2-root/instance:
uuid: "f069c078cd8c49e68c86e6f87ffcab75"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.069938 23922 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.072329 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.002s
I1014 19:25:35.073997 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.076596 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I1014 19:25:35.076699 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-2-root
uuid: "f069c078cd8c49e68c86e6f87ffcab75"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.076913 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-2-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-2-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.078953 23922 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:35.080232 23922 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:55302: Network error: Client connection negotiation failed: client connection to 127.20.201.60:55302: connect: Connection refused (error 111)
I1014 19:25:35.111506 23922 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } attempt: 1
W1014 19:25:35.112444 23922 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:55302: Network error: Client connection negotiation failed: client connection to 127.20.201.60:55302: connect: Connection refused (error 111)
I1014 19:25:35.133838 23790 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } attempt: 3
I1014 19:25:35.139073 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:55302
I1014 19:25:35.139380 21284 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:35.139451 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.140357 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:40302/
I1014 19:25:35.140807 23790 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.141038 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0
I1014 19:25:35.143118 23980 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:35.146441 23980 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.150202 23790 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Bootstrap starting.
I1014 19:25:35.151839 23790 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: No blocks or log segments found. Creating new log.
I1014 19:25:35.153463 23980 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.154916 23790 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: No bootstrap required, opened a new log
I1014 19:25:35.156003 23790 raft_consensus.cc:325] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.156294 23790 raft_consensus.cc:351] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.156390 23790 raft_consensus.cc:630] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2aeda8be7c4b4b8cf586054b37c2fe, State: Initialized, Role: FOLLOWER
I1014 19:25:35.156714 23790 consensus_queue.cc:228] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.157317 23790 sys_catalog.cc:337] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.157718 23790 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.157727 23984 sys_catalog.cc:337] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.158321 23984 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.158568 23790 sys_catalog.cc:424] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:35.159682 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1
I1014 19:25:35.159672 23980 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:35.163425 23985 catalog_manager.cc:1084] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:35.163568 23985 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.166237 23980 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: Bootstrap starting.
I1014 19:25:35.167534 23980 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: No blocks or log segments found. Creating new log.
I1014 19:25:35.169255 23980 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: No bootstrap required, opened a new log
I1014 19:25:35.169867 23980 raft_consensus.cc:325] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.170032 23980 raft_consensus.cc:351] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.170084 23980 raft_consensus.cc:630] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f069c078cd8c49e68c86e6f87ffcab75, State: Initialized, Role: FOLLOWER
I1014 19:25:35.170301 23980 consensus_queue.cc:228] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.170789 23980 sys_catalog.cc:337] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.171134 23980 sys_catalog.cc:340] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.171015 23987 sys_catalog.cc:337] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.171447 23987 sys_catalog.cc:340] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.171697 23980 sys_catalog.cc:424] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: configured and running, proceeding with master startup.
W1014 19:25:35.172936 23988 catalog_manager.cc:1084] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:35.172993 23988 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.185660 23922 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } attempt: 2
I1014 19:25:35.191869 23922 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: Bootstrap starting.
I1014 19:25:35.193047 23922 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: No blocks or log segments found. Creating new log.
I1014 19:25:35.194883 23922 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: No bootstrap required, opened a new log
I1014 19:25:35.195508 23922 raft_consensus.cc:325] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.195693 23922 raft_consensus.cc:351] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.195747 23922 raft_consensus.cc:630] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3fc2ca813e4055b02ec0351fa5a70c, State: Initialized, Role: FOLLOWER
I1014 19:25:35.195983 23922 consensus_queue.cc:228] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.196513 23922 sys_catalog.cc:337] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.196813 23922 sys_catalog.cc:340] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.196652 23990 sys_catalog.cc:337] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.197192 23990 sys_catalog.cc:340] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.197474 23922 sys_catalog.cc:424] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:35.197818 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2
W1014 19:25:35.202281 23991 catalog_manager.cc:1084] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:35.202445 23991 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.202965 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
I1014 19:25:35.203138 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.203199 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.203301 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.203351 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.203871 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/ts-0-root/instance:
uuid: "71c1fa97bb704b6aa1cb2bd1f5763101"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.208120 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I1014 19:25:35.208878 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.211460 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I1014 19:25:35.211541 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/ts-0-root
uuid: "71c1fa97bb704b6aa1cb2bd1f5763101"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.211637 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/ts-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/ts-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.RefuseToSignInvalidCSR.1539570329182132-21284/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.244489 23984 raft_consensus.cc:457] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:35.244664 23984 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Snoozing failure detection for 2.200s (starting election)
I1014 19:25:35.244812 23984 raft_consensus.cc:479] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.245393 23984 leader_election.cc:251] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ec3fc2ca813e4055b02ec0351fa5a70c (127.20.201.61:44802)
I1014 19:25:35.245766 23984 leader_election.cc:251] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f069c078cd8c49e68c86e6f87ffcab75 (127.20.201.60:55302)
I1014 19:25:35.246424 23887 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" is_pre_election: true
I1014 19:25:35.246768 23887 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: Snoozing failure detection for 1.788s (vote granted)
I1014 19:25:35.246680 23957 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f069c078cd8c49e68c86e6f87ffcab75" is_pre_election: true
I1014 19:25:35.246992 23957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: Snoozing failure detection for 1.844s (vote granted)
I1014 19:25:35.246949 23887 raft_consensus.cc:2255] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2aeda8be7c4b4b8cf586054b37c2fe in term 0.
I1014 19:25:35.247122 23957 raft_consensus.cc:2255] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2aeda8be7c4b4b8cf586054b37c2fe in term 0.
I1014 19:25:35.247498 23704 leader_election.cc:387] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 pre-election: Vote granted by peer ec3fc2ca813e4055b02ec0351fa5a70c (127.20.201.61:44802)
I1014 19:25:35.247604 23704 leader_election.cc:278] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I1014 19:25:35.247992 23702 leader_election.cc:387] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 pre-election: Vote granted by peer f069c078cd8c49e68c86e6f87ffcab75 (127.20.201.60:55302)
I1014 19:25:35.247994 23984 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Snoozing failure detection for 2.142s (election complete)
I1014 19:25:35.248149 23984 raft_consensus.cc:2549] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I1014 19:25:35.248252 23984 raft_consensus.cc:457] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:35.248379 23984 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Snoozing failure detection for 1.635s (starting election)
I1014 19:25:35.248500 23984 raft_consensus.cc:2795] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.251785 23984 raft_consensus.cc:479] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.252315 23984 leader_election.cc:251] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 election: Requesting vote from peer ec3fc2ca813e4055b02ec0351fa5a70c (127.20.201.61:44802)
I1014 19:25:35.252610 23984 leader_election.cc:251] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 election: Requesting vote from peer f069c078cd8c49e68c86e6f87ffcab75 (127.20.201.60:55302)
I1014 19:25:35.253195 23887 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c"
I1014 19:25:35.253193 23957 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f069c078cd8c49e68c86e6f87ffcab75"
I1014 19:25:35.253445 23957 raft_consensus.cc:2795] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.253588 23957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: Snoozing failure detection for 1.919s (vote granted)
I1014 19:25:35.253445 23887 raft_consensus.cc:2795] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.253739 23887 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: Snoozing failure detection for 1.680s (vote granted)
I1014 19:25:35.256513 23887 raft_consensus.cc:2255] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2aeda8be7c4b4b8cf586054b37c2fe in term 1.
I1014 19:25:35.256512 23957 raft_consensus.cc:2255] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2aeda8be7c4b4b8cf586054b37c2fe in term 1.
I1014 19:25:35.256889 23704 leader_election.cc:387] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 election: Vote granted by peer ec3fc2ca813e4055b02ec0351fa5a70c (127.20.201.61:44802)
I1014 19:25:35.256994 23704 leader_election.cc:278] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:35.257113 23702 leader_election.cc:387] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [CANDIDATE]: Term 1 election: Vote granted by peer f069c078cd8c49e68c86e6f87ffcab75 (127.20.201.60:55302)
I1014 19:25:35.257445 23984 raft_consensus.cc:2743] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Snoozing failure detection for 1.679s (election complete)
I1014 19:25:35.257638 23984 raft_consensus.cc:2549] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 1 FOLLOWER]: Leader election won for term 1
I1014 19:25:35.258329 23984 raft_consensus.cc:595] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 1 LEADER]: Becoming Leader. State: Replica: ef2aeda8be7c4b4b8cf586054b37c2fe, State: Running, Role: LEADER
I1014 19:25:35.258708 23984 consensus_queue.cc:205] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } }
I1014 19:25:35.260327 23998 sys_catalog.cc:337] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2aeda8be7c4b4b8cf586054b37c2fe. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.260798 23998 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.265821 24000 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:35.267211 24000 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:35.301983 21284 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:35.320585 23957 raft_consensus.cc:1083] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 1 FOLLOWER]: Refusing update from remote peer ef2aeda8be7c4b4b8cf586054b37c2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:35.320731 23887 raft_consensus.cc:1083] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 1 FOLLOWER]: Refusing update from remote peer ef2aeda8be7c4b4b8cf586054b37c2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:35.321316 23998 consensus_queue.cc:963] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:35.321650 23984 consensus_queue.cc:963] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:35.326903 23987 sys_catalog.cc:337] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2aeda8be7c4b4b8cf586054b37c2fe. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.327209 23987 sys_catalog.cc:340] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.329463 23990 sys_catalog.cc:337] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2aeda8be7c4b4b8cf586054b37c2fe. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.330570 23990 sys_catalog.cc:340] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.335850 23984 sys_catalog.cc:337] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.336407 23984 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.337007 24000 catalog_manager.cc:924] Generated new certificate authority record
I1014 19:25:35.337070 23987 sys_catalog.cc:337] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.337844 23987 sys_catalog.cc:340] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.339200 24000 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:35.340559 23984 sys_catalog.cc:337] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.340858 23984 sys_catalog.cc:340] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.350260 23990 sys_catalog.cc:337] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2aeda8be7c4b4b8cf586054b37c2fe" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 50565 } } peers { permanent_uuid: "ec3fc2ca813e4055b02ec0351fa5a70c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 44802 } } peers { permanent_uuid: "f069c078cd8c49e68c86e6f87ffcab75" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 55302 } } }
I1014 19:25:35.350620 23990 sys_catalog.cc:340] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.354445 24000 catalog_manager.cc:4062] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: Generated new TSK 0
I1014 19:25:35.401860 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:48110
I1014 19:25:35.402109 21284 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:35.402164 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.402839 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:40810/
I1014 19:25:35.421722 24109 heartbeater.cc:346] Connected to a master server at 127.20.201.62:50565
I1014 19:25:35.421901 24109 heartbeater.cc:426] Registering TS with master...
I1014 19:25:35.422508 24109 heartbeater.cc:475] Master 127.20.201.62:50565 requested a full tablet report, sending...
I1014 19:25:35.424299 23737 ts_manager.cc:98] Registered new tserver with Master: 71c1fa97bb704b6aa1cb2bd1f5763101 (127.20.201.1:48110)
I1014 19:25:35.426635 24119 heartbeater.cc:346] Connected to a master server at 127.20.201.60:55302
I1014 19:25:35.426640 23737 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.62:36713
I1014 19:25:35.426744 24119 heartbeater.cc:426] Registering TS with master...
I1014 19:25:35.427248 24119 heartbeater.cc:475] Master 127.20.201.60:55302 requested a full tablet report, sending...
I1014 19:25:35.428009 24110 heartbeater.cc:346] Connected to a master server at 127.20.201.61:44802
I1014 19:25:35.428153 24110 heartbeater.cc:426] Registering TS with master...
I1014 19:25:35.429049 24110 heartbeater.cc:475] Master 127.20.201.61:44802 requested a full tablet report, sending...
I1014 19:25:35.429441 23947 ts_manager.cc:98] Registered new tserver with Master: 71c1fa97bb704b6aa1cb2bd1f5763101 (127.20.201.1:48110)
I1014 19:25:35.430465 23871 ts_manager.cc:98] Registered new tserver with Master: 71c1fa97bb704b6aa1cb2bd1f5763101 (127.20.201.1:48110)
I1014 19:25:35.430865 21284 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.01383635s
I1014 19:25:35.441154 23737 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:35.450036 23871 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:35.455915 23947 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:35.468497 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:35.480511 21284 ts_tablet_manager.cc:1118] Shutting down tablet manager...
I1014 19:25:35.494168 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:35.494236 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:35.494419 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:35.497334 21284 master.cc:249] Master@127.20.201.62:50565 shutting down...
I1014 19:25:35.505759 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 1 LEADER]: Raft consensus shutting down.
I1014 19:25:35.505836 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe -> Peer f069c078cd8c49e68c86e6f87ffcab75 (127.20.201.60:55302): Closing peer: f069c078cd8c49e68c86e6f87ffcab75
I1014 19:25:35.505921 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe -> Peer ec3fc2ca813e4055b02ec0351fa5a70c (127.20.201.61:44802): Closing peer: ec3fc2ca813e4055b02ec0351fa5a70c
I1014 19:25:35.506137 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:35.506398 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P ef2aeda8be7c4b4b8cf586054b37c2fe: stopping tablet replica
I1014 19:25:35.506659 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:35.506731 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.506798 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.506861 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:35.507308 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:35.507360 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:35.507414 21284 maintenance_manager.cc:237] P ef2aeda8be7c4b4b8cf586054b37c2fe: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:35.521417 21284 master.cc:260] Master@127.20.201.62:50565 shutdown complete.
I1014 19:25:35.524202 21284 master.cc:249] Master@127.20.201.61:44802 shutting down...
I1014 19:25:35.532805 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 1 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:35.533010 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:35.533138 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P ec3fc2ca813e4055b02ec0351fa5a70c: stopping tablet replica
I1014 19:25:35.533262 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:35.533356 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.533447 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.533557 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:35.533655 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:35.533727 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:35.533802 21284 maintenance_manager.cc:237] P ec3fc2ca813e4055b02ec0351fa5a70c: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:35.538691 21284 master.cc:260] Master@127.20.201.61:44802 shutdown complete.
I1014 19:25:35.542004 21284 master.cc:249] Master@127.20.201.60:55302 shutting down...
I1014 19:25:35.551606 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 1 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:35.551748 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75 [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:35.551838 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P f069c078cd8c49e68c86e6f87ffcab75: stopping tablet replica
I1014 19:25:35.551926 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:35.551990 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.552052 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:35.552109 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:35.552176 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:35.552224 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:35.552276 21284 maintenance_manager.cc:237] P f069c078cd8c49e68c86e6f87ffcab75: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:35.557672 21284 master.cc:260] Master@127.20.201.60:55302 shutdown complete.
[       OK ] MasterCertAuthorityTest.RefuseToSignInvalidCSR (697 ms)
[ RUN      ] MasterCertAuthorityTest.MasterLeaderSignsCSR
I1014 19:25:35.566915 21284 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.20.201.62:41315,127.20.201.61:57943,127.20.201.60:40754
I1014 19:25:35.571880 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
I1014 19:25:35.572079 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.572131 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.572268 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.572333 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.573019 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-0-root/instance:
uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.577867 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I1014 19:25:35.578857 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.581550 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
I1014 19:25:35.581641 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-0-root
uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.581806 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.630744 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:41315
I1014 19:25:35.630972 21284 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:35.631012 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.631544 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:59948/
I1014 19:25:35.632774 24259 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:35.635675 24259 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.636960 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
I1014 19:25:35.637326 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.637542 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.637755 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.638059 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.638638 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-1-root/instance:
uuid: "c44cdb4edc6148b4a275d1baa06ba98c"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.643084 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.004s
I1014 19:25:35.644227 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.645413 24259 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:35.646404 24259 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:57943: Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:35.647063 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.002s
I1014 19:25:35.647130 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-1-root
uuid: "c44cdb4edc6148b4a275d1baa06ba98c"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.647225 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-1-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-1-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.672673 24259 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } attempt: 1
W1014 19:25:35.673781 24259 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:57943: Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:35.708566 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:57943
I1014 19:25:35.708838 21284 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:35.708876 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.709460 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:46250/
I1014 19:25:35.710688 24316 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:35.714009 24316 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.715646 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
I1014 19:25:35.715863 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.715943 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.716089 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.716177 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.716735 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-2-root/instance:
uuid: "d223c42c215145f4ab6ec86b332ac9f1"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.721848 24316 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.721968 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.001s
I1014 19:25:35.723731 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.727599 21284 fs_manager.cc:419] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
I1014 19:25:35.727748 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-2-root
uuid: "d223c42c215145f4ab6ec86b332ac9f1"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.727923 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-2-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-2-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.729254 24316 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:35.730284 24316 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:40754: Network error: Client connection negotiation failed: client connection to 127.20.201.60:40754: connect: Connection refused (error 111)
I1014 19:25:35.734005 24259 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } attempt: 2
I1014 19:25:35.739461 24259 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:35.740447 24259 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:40754: Network error: Client connection negotiation failed: client connection to 127.20.201.60:40754: connect: Connection refused (error 111)
I1014 19:25:35.765542 24316 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } attempt: 1
W1014 19:25:35.766543 24316 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:40754: Network error: Client connection negotiation failed: client connection to 127.20.201.60:40754: connect: Connection refused (error 111)
I1014 19:25:35.779681 24259 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } attempt: 1
W1014 19:25:35.780450 24259 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:40754: Network error: Client connection negotiation failed: client connection to 127.20.201.60:40754: connect: Connection refused (error 111)
I1014 19:25:35.815346 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:40754
I1014 19:25:35.815618 21284 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:35.815668 21284 webserver.cc:186] Document root disabled
I1014 19:25:35.816221 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:49169/
I1014 19:25:35.816980 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0
I1014 19:25:35.817580 24394 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:35.819674 24394 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.824882 24394 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.829318 24394 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:35.836848 24394 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Bootstrap starting.
I1014 19:25:35.838069 24394 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: No blocks or log segments found. Creating new log.
I1014 19:25:35.840229 24394 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: No bootstrap required, opened a new log
I1014 19:25:35.840909 24394 raft_consensus.cc:325] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.841101 24394 raft_consensus.cc:351] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.841150 24394 raft_consensus.cc:630] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d223c42c215145f4ab6ec86b332ac9f1, State: Initialized, Role: FOLLOWER
I1014 19:25:35.841337 24394 consensus_queue.cc:228] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.841861 24394 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.842171 24394 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.841941 24413 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.842330 24413 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.842679 24394 sys_catalog.cc:424] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: configured and running, proceeding with master startup.
W1014 19:25:35.843854 24416 catalog_manager.cc:1084] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:35.843911 24416 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.844760 24316 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } attempt: 2
I1014 19:25:35.850723 24316 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Bootstrap starting.
I1014 19:25:35.851647 24316 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: No blocks or log segments found. Creating new log.
I1014 19:25:35.853323 24316 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: No bootstrap required, opened a new log
I1014 19:25:35.853699 24259 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } attempt: 2
I1014 19:25:35.854154 24316 raft_consensus.cc:325] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.854410 24316 raft_consensus.cc:351] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.854482 24316 raft_consensus.cc:630] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44cdb4edc6148b4a275d1baa06ba98c, State: Initialized, Role: FOLLOWER
I1014 19:25:35.855012 24316 consensus_queue.cc:228] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.856806 24316 sys_catalog.cc:337] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.857228 24316 sys_catalog.cc:340] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.857753 24316 sys_catalog.cc:424] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:35.856940 24422 sys_catalog.cc:337] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.861549 24422 sys_catalog.cc:340] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:25:35.861656 24424 catalog_manager.cc:1084] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:35.861728 24424 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.862381 24259 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Bootstrap starting.
I1014 19:25:35.863638 24259 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: No blocks or log segments found. Creating new log.
I1014 19:25:35.865201 24422 raft_consensus.cc:457] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I1014 19:25:35.865350 24422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Snoozing failure detection for 2.131s (starting election)
I1014 19:25:35.865537 24259 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: No bootstrap required, opened a new log
I1014 19:25:35.865545 24422 raft_consensus.cc:479] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.866113 24422 leader_election.cc:251] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:35.866372 24422 leader_election.cc:251] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:35.866544 24259 raft_consensus.cc:325] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.866773 24259 raft_consensus.cc:351] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:35.866866 24259 raft_consensus.cc:630] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5f29eaf82747b5919e7d5a0bb0e09f, State: Initialized, Role: FOLLOWER
I1014 19:25:35.867130 24259 consensus_queue.cc:228] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.867131 24360 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d223c42c215145f4ab6ec86b332ac9f1" is_pre_election: true
I1014 19:25:35.867713 24360 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 1.720s (vote granted)
I1014 19:25:35.867835 24360 raft_consensus.cc:2255] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c44cdb4edc6148b4a275d1baa06ba98c in term 0.
I1014 19:25:35.868435 24266 leader_election.cc:387] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 pre-election: Vote granted by peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:35.868628 24266 leader_election.cc:278] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.
I1014 19:25:35.868620 24259 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.869025 24422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Snoozing failure detection for 2.111s (election complete)
I1014 19:25:35.869176 24422 raft_consensus.cc:2549] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I1014 19:25:35.869276 24422 raft_consensus.cc:457] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:35.869333 24259 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.869613 24422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Snoozing failure detection for 1.535s (starting election)
I1014 19:25:35.869714 24422 raft_consensus.cc:2795] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.869021 24432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.869953 24432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.871526 24422 raft_consensus.cc:479] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.872084 24259 sys_catalog.cc:424] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:35.872086 24422 leader_election.cc:251] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 election: Requesting vote from peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:35.872090 24236 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" is_pre_election: true
I1014 19:25:35.872396 24422 leader_election.cc:251] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 election: Requesting vote from peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:35.872402 24236 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 1.598s (vote granted)
I1014 19:25:35.872740 24236 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c44cdb4edc6148b4a275d1baa06ba98c in term 0.
I1014 19:25:35.872903 24360 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d223c42c215145f4ab6ec86b332ac9f1"
I1014 19:25:35.873059 24360 raft_consensus.cc:2795] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.873283 24360 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 2.244s (vote granted)
I1014 19:25:35.873286 24265 leader_election.cc:387] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:35.874969 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 1
I1014 19:25:35.875207 21284 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 2
I1014 19:25:35.875207 24235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f"
I1014 19:25:35.875494 24235 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:35.875620 24235 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 1.752s (vote granted)
I1014 19:25:35.879325 24360 raft_consensus.cc:2255] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c44cdb4edc6148b4a275d1baa06ba98c in term 1.
I1014 19:25:35.879483 24235 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c44cdb4edc6148b4a275d1baa06ba98c in term 1.
I1014 19:25:35.879782 24265 leader_election.cc:387] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 election: Vote granted by peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:35.879984 24265 leader_election.cc:278] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:35.882241 24422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Snoozing failure detection for 1.789s (election complete)
I1014 19:25:35.882247 24266 leader_election.cc:387] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [CANDIDATE]: Term 1 election: Vote granted by peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:35.882376 24422 raft_consensus.cc:2549] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 1 FOLLOWER]: Leader election won for term 1
W1014 19:25:35.882527 24438 catalog_manager.cc:1084] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: acquiring CA information for follower catalog manager: Not found: root CA entry not found
I1014 19:25:35.883153 21284 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 420078us
W1014 19:25:35.883461 24438 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:35.883757 21284 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:35.883882 21284 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:35.884068 21284 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:35.884158 21284 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:35.884847 21284 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/ts-0-root/instance:
uuid: "e4b831eed6914352a438481857f878af"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.888356 24422 raft_consensus.cc:595] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 1 LEADER]: Becoming Leader. State: Replica: c44cdb4edc6148b4a275d1baa06ba98c, State: Running, Role: LEADER
I1014 19:25:35.888725 24422 consensus_queue.cc:205] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:35.891082 21284 fs_manager.cc:503] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.003s
I1014 19:25:35.892812 21284 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:35.895750 24442 sys_catalog.cc:337] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44cdb4edc6148b4a275d1baa06ba98c. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.896091 24442 sys_catalog.cc:340] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.897207 24447 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:35.899441 24447 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:35.902556 21284 fs_manager.cc:419] Time spent opening block manager: real 0.007s	user 0.000s	sys 0.005s
I1014 19:25:35.902694 21284 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/ts-0-root
uuid: "e4b831eed6914352a438481857f878af"
format_stamp: "Formatted at 2018-10-15 02:25:35 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:35.902817 21284 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/ts-0-root
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/ts-0-root
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MasterCertAuthorityTest.MasterLeaderSignsCSR.1539570329182132-21284/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:35.932368 24360 raft_consensus.cc:1083] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Refusing update from remote peer c44cdb4edc6148b4a275d1baa06ba98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:35.932849 24235 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Refusing update from remote peer c44cdb4edc6148b4a275d1baa06ba98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I1014 19:25:35.933552 24442 consensus_queue.cc:963] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:35.933773 24422 consensus_queue.cc:963] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:35.944237 24432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44cdb4edc6148b4a275d1baa06ba98c. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.944676 24432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.946946 24413 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c44cdb4edc6148b4a275d1baa06ba98c. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.947257 24413 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.952373 24442 sys_catalog.cc:337] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.953296 24442 sys_catalog.cc:340] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.953481 24442 sys_catalog.cc:337] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.957486 24442 sys_catalog.cc:340] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [sys.catalog]: This master's current role is: LEADER
I1014 19:25:35.958690 24447 catalog_manager.cc:924] Generated new certificate authority record
I1014 19:25:35.959606 24413 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.960639 24413 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.962988 24447 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:35.963179 24432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:35.963524 24432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:35.968503 21284 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:35.973800 24447 catalog_manager.cc:4062] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: Generated new TSK 0
I1014 19:25:36.056486 21284 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:53104
I1014 19:25:36.056850 21284 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:36.056893 21284 webserver.cc:186] Document root disabled
I1014 19:25:36.057648 21284 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:59497/
I1014 19:25:36.083772 24584 heartbeater.cc:346] Connected to a master server at 127.20.201.62:41315
I1014 19:25:36.083902 24584 heartbeater.cc:426] Registering TS with master...
I1014 19:25:36.084988 24584 heartbeater.cc:475] Master 127.20.201.62:41315 requested a full tablet report, sending...
I1014 19:25:36.086833 24226 ts_manager.cc:98] Registered new tserver with Master: e4b831eed6914352a438481857f878af (127.20.201.1:53104)
I1014 19:25:36.089787 24587 heartbeater.cc:346] Connected to a master server at 127.20.201.60:40754
I1014 19:25:36.089906 24587 heartbeater.cc:426] Registering TS with master...
I1014 19:25:36.090412 24587 heartbeater.cc:475] Master 127.20.201.60:40754 requested a full tablet report, sending...
I1014 19:25:36.091249 24585 heartbeater.cc:346] Connected to a master server at 127.20.201.61:57943
I1014 19:25:36.091361 24585 heartbeater.cc:426] Registering TS with master...
I1014 19:25:36.091657 24350 ts_manager.cc:98] Registered new tserver with Master: e4b831eed6914352a438481857f878af (127.20.201.1:53104)
I1014 19:25:36.091831 24585 heartbeater.cc:475] Master 127.20.201.61:57943 requested a full tablet report, sending...
I1014 19:25:36.092603 24283 ts_manager.cc:98] Registered new tserver with Master: e4b831eed6914352a438481857f878af (127.20.201.1:53104)
I1014 19:25:36.093298 21284 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.007720847s
I1014 19:25:36.093878 24283 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.61:55015
I1014 19:25:36.102030 24226 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:36.106611 24283 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:36.111855 24350 ts_manager.cc:98] Registered new tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:36.117475 24283 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.61:55018
I1014 19:25:36.119541 21284 master.cc:249] Master@127.20.201.61:57943 shutting down...
I1014 19:25:36.130290 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 1 LEADER]: Raft consensus shutting down.
I1014 19:25:36.130375 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c -> Peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754): Closing peer: d223c42c215145f4ab6ec86b332ac9f1
I1014 19:25:36.130463 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c -> Peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315): Closing peer: 7c5f29eaf82747b5919e7d5a0bb0e09f
I1014 19:25:36.130697 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c [term 1 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:36.130908 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P c44cdb4edc6148b4a275d1baa06ba98c: stopping tablet replica
I1014 19:25:36.131119 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:36.131204 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:36.131287 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:36.131345 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:36.131420 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:36.131471 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:36.131536 21284 maintenance_manager.cc:237] P c44cdb4edc6148b4a275d1baa06ba98c: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:36.136519 21284 master.cc:260] Master@127.20.201.61:57943 shutdown complete.
I1014 19:25:36.846684 24416 catalog_manager.cc:1082] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: acquiring CA information for follower catalog manager: success
I1014 19:25:36.847674 24416 catalog_manager.cc:1110] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:36.886579 24438 catalog_manager.cc:1082] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: acquiring CA information for follower catalog manager: success
I1014 19:25:36.887502 24438 catalog_manager.cc:1110] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:37.096010 24585 heartbeater.cc:467] Master 127.20.201.61:57943 was elected leader, sending a full tablet report...
W1014 19:25:37.107388 24585 heartbeater.cc:587] Failed to heartbeat to 127.20.201.61:57943: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:37.480059 25095 raft_consensus.cc:457] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c44cdb4edc6148b4a275d1baa06ba98c)
I1014 19:25:37.480173 25095 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 1.784s (starting election)
I1014 19:25:37.480312 25095 raft_consensus.cc:479] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:37.481045 25095 leader_election.cc:251] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:37.481498 25095 leader_election.cc:251] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943)
I1014 19:25:37.482095 24235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d223c42c215145f4ab6ec86b332ac9f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" is_pre_election: true
I1014 19:25:37.482306 24235 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 1.774s (vote granted)
I1014 19:25:37.482520 24235 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d223c42c215145f4ab6ec86b332ac9f1 in term 1.
I1014 19:25:37.483116 25096 raft_consensus.cc:457] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c44cdb4edc6148b4a275d1baa06ba98c)
I1014 19:25:37.483260 25096 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 1.964s (starting election)
I1014 19:25:37.483436 25096 raft_consensus.cc:479] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:37.483928 25096 leader_election.cc:251] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943)
I1014 19:25:37.484161 25096 leader_election.cc:251] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:37.485100 24325 leader_election.cc:387] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:37.485200 24325 leader_election.cc:278] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I1014 19:25:37.485579 25095 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 1.533s (election complete)
I1014 19:25:37.485651 24360 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d223c42c215145f4ab6ec86b332ac9f1" is_pre_election: true
I1014 19:25:37.485738 25095 raft_consensus.cc:2549] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I1014 19:25:37.485865 24360 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 1.516s (vote granted)
I1014 19:25:37.485975 24360 raft_consensus.cc:2255] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c5f29eaf82747b5919e7d5a0bb0e09f in term 1.
I1014 19:25:37.486178 25095 raft_consensus.cc:457] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c44cdb4edc6148b4a275d1baa06ba98c)
W1014 19:25:37.485579 24325 leader_election.cc:307] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943): Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:37.486322 25095 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 1.630s (starting election)
I1014 19:25:37.486572 25095 raft_consensus.cc:2795] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 1 FOLLOWER]: Advancing to term 2
W1014 19:25:37.487964 24208 leader_election.cc:307] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943): Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:37.489205 25095 raft_consensus.cc:479] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:37.489629 25095 leader_election.cc:251] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 election: Requesting vote from peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:37.489822 25095 leader_election.cc:251] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 election: Requesting vote from peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943)
I1014 19:25:37.490100 24209 leader_election.cc:387] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [CANDIDATE]: Term 2 pre-election: Vote granted by peer d223c42c215145f4ab6ec86b332ac9f1 (127.20.201.60:40754)
I1014 19:25:37.490200 24209 leader_election.cc:278] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won.
I1014 19:25:37.490198 24235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d223c42c215145f4ab6ec86b332ac9f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f"
I1014 19:25:37.490453 24235 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:37.490530 24235 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 2.243s (vote granted)
I1014 19:25:37.492342 24235 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d223c42c215145f4ab6ec86b332ac9f1 in term 2.
I1014 19:25:37.492524 25096 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: Snoozing failure detection for 1.811s (election complete)
I1014 19:25:37.492668 25096 raft_consensus.cc:2512] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I1014 19:25:37.493021 24325 leader_election.cc:387] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 election: Vote granted by peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315)
I1014 19:25:37.493144 24325 leader_election.cc:278] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
W1014 19:25:37.493394 24325 leader_election.cc:307] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943): Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111)
I1014 19:25:37.493394 25095 raft_consensus.cc:2743] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Snoozing failure detection for 2.148s (election complete)
I1014 19:25:37.493692 25095 raft_consensus.cc:2549] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 2 FOLLOWER]: Leader election won for term 2
I1014 19:25:37.494103 25095 raft_consensus.cc:595] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 2 LEADER]: Becoming Leader. State: Replica: d223c42c215145f4ab6ec86b332ac9f1, State: Running, Role: LEADER
I1014 19:25:37.494480 25095 consensus_queue.cc:205] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } }
I1014 19:25:37.495824 25099 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d223c42c215145f4ab6ec86b332ac9f1. Latest consensus state: current_term: 2 leader_uuid: "d223c42c215145f4ab6ec86b332ac9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:37.496057 25099 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:37.496683 25101 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:37.498888 25101 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:37.500320 25101 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:37.501521 25101 catalog_manager.cc:4073] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: Loaded TSK: 0
W1014 19:25:37.505492 24325 consensus_peers.cc:433] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 -> Peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943): Couldn't send request to peer c44cdb4edc6148b4a275d1baa06ba98c for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.20.201.61:57943: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times.
I1014 19:25:37.505776 24235 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 2 FOLLOWER]: Refusing update from remote peer d223c42c215145f4ab6ec86b332ac9f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I1014 19:25:37.506271 25099 consensus_queue.cc:963] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I1014 19:25:37.508348 25096 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d223c42c215145f4ab6ec86b332ac9f1. Latest consensus state: current_term: 2 leader_uuid: "d223c42c215145f4ab6ec86b332ac9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:37.508692 25096 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:37.509809 25099 sys_catalog.cc:337] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d223c42c215145f4ab6ec86b332ac9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:37.510593 25099 sys_catalog.cc:340] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:37.511708 25096 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d223c42c215145f4ab6ec86b332ac9f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c5f29eaf82747b5919e7d5a0bb0e09f" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 41315 } } peers { permanent_uuid: "c44cdb4edc6148b4a275d1baa06ba98c" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 57943 } } peers { permanent_uuid: "d223c42c215145f4ab6ec86b332ac9f1" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 40754 } } }
I1014 19:25:37.511926 25096 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:37.549309 24226 ts_descriptor.cc:209] Processing retry of TS registration from permanent_uuid: "fake-ts-uuid" instance_seqno: 1
I1014 19:25:37.549434 24226 ts_manager.cc:104] Re-registered known tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:37.550640 24350 ts_descriptor.cc:209] Processing retry of TS registration from permanent_uuid: "fake-ts-uuid" instance_seqno: 1
I1014 19:25:37.550727 24350 ts_manager.cc:104] Re-registered known tserver with Master: fake-ts-uuid (localhost:1000)
I1014 19:25:37.554502 24350 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.60:53075
I1014 19:25:37.555814 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:37.568214 21284 ts_tablet_manager.cc:1118] Shutting down tablet manager...
I1014 19:25:37.581106 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:37.581156 21284 tablet_server.cc:146] TabletServer shutting down...
I1014 19:25:37.581318 21284 tablet_server.cc:160] TabletServer shutdown complete.
I1014 19:25:37.583812 21284 master.cc:249] Master@127.20.201.62:41315 shutting down...
I1014 19:25:37.594627 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 2 FOLLOWER]: Raft consensus shutting down.
I1014 19:25:37.594772 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:37.594884 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P 7c5f29eaf82747b5919e7d5a0bb0e09f: stopping tablet replica
I1014 19:25:37.594970 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:37.595062 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:37.595125 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:37.595185 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:37.595252 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:37.595304 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:37.595351 21284 maintenance_manager.cc:237] P 7c5f29eaf82747b5919e7d5a0bb0e09f: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:37.609515 21284 master.cc:260] Master@127.20.201.62:41315 shutdown complete.
I1014 19:25:37.612110 21284 master.cc:249] Master@127.20.201.60:40754 shutting down...
I1014 19:25:37.621506 21284 raft_consensus.cc:2052] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 2 LEADER]: Raft consensus shutting down.
I1014 19:25:37.621575 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 -> Peer c44cdb4edc6148b4a275d1baa06ba98c (127.20.201.61:57943): Closing peer: c44cdb4edc6148b4a275d1baa06ba98c
I1014 19:25:37.621659 21284 consensus_peers.cc:455] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 -> Peer 7c5f29eaf82747b5919e7d5a0bb0e09f (127.20.201.62:41315): Closing peer: 7c5f29eaf82747b5919e7d5a0bb0e09f
I1014 19:25:37.621841 21284 raft_consensus.cc:2079] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I1014 19:25:37.622117 21284 tablet_replica.cc:263] T 00000000000000000000000000000000 P d223c42c215145f4ab6ec86b332ac9f1: stopping tablet replica
I1014 19:25:37.622305 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op CompactRowSetsOp(00000000000000000000000000000000)
I1014 19:25:37.622380 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:37.622462 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000)
I1014 19:25:37.622524 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op UndoDeltaBlockGCOp(00000000000000000000000000000000)
I1014 19:25:37.622589 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op FlushMRSOp(00000000000000000000000000000000)
I1014 19:25:37.622642 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000)
I1014 19:25:37.622689 21284 maintenance_manager.cc:237] P d223c42c215145f4ab6ec86b332ac9f1: Unregistered op LogGCOp(00000000000000000000000000000000)
I1014 19:25:37.636134 21284 master.cc:260] Master@127.20.201.60:40754 shutdown complete.
[       OK ] MasterCertAuthorityTest.MasterLeaderSignsCSR (2076 ms)
[----------] 4 tests from MasterCertAuthorityTest (8409 ms total)

[----------] 1 test from SingleMasterConnectToClusterTest
[ RUN      ] SingleMasterConnectToClusterTest.ConnectToCluster
I1014 19:25:37.644803 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.62:36022
--webserver_interface=127.20.201.62
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--catalog_manager_inject_latency_load_ca_info_ms=2500
--rpc_reuseport=true with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:37.854739 25118 flags.cc:406] Enabled unsafe flag: --catalog_manager_inject_latency_load_ca_info_ms=2500
W1014 19:25:37.854957 25118 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:37.855049 25118 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W1014 19:25:37.857461 25118 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W1014 19:25:37.857547 25118 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:37.857594 25118 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W1014 19:25:37.857655 25118 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I1014 19:25:37.862093 25118 master_main.cc:73] Master server non-default flags:
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
--catalog_manager_inject_latency_load_ca_info_ms=2500
--ipki_ca_key_size=1024
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.62:36022
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.201.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
I1014 19:25:37.867985 25118 master_main.cc:80] Initializing master server...
I1014 19:25:37.868777 25118 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 421078us
I1014 19:25:37.869925 25118 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:37.870003 25118 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:37.870267 25118 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:37.870345 25118 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:37.872864 25118 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/instance:
uuid: "f000641d996c4cdba3b4481a74eda150"
format_stamp: "Formatted at 2018-10-15 02:25:37 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:37.873438 25118 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal/instance:
uuid: "f000641d996c4cdba3b4481a74eda150"
format_stamp: "Formatted at 2018-10-15 02:25:37 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:37.881268 25118 fs_manager.cc:503] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.001s
I1014 19:25:37.882272 25118 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:37.882791 25118 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:37.886330 25118 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I1014 19:25:37.886390 25118 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
uuid: "f000641d996c4cdba3b4481a74eda150"
format_stamp: "Formatted at 2018-10-15 02:25:37 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:37.886579 25118 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:37.947598 25118 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:37.948674 25118 master_main.cc:83] Starting Master server...
I1014 19:25:37.990677 25118 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:36022
I1014 19:25:37.991173 25118 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:37.991214 25118 webserver.cc:186] Document root disabled
I1014 19:25:37.991978 25118 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:37705/
I1014 19:25:37.992756 25118 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
I1014 19:25:37.997463 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 25118
I1014 19:25:37.998447 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.20.201.1:0
--local_ip_for_outbound_sockets=127.20.201.1
--webserver_interface=127.20.201.1
--webserver_port=0
--tserver_master_addrs=127.20.201.62:36022
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags with env {}
I1014 19:25:38.000787 25324 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:38.013209 25324 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Bootstrap starting.
I1014 19:25:38.016106 25324 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: No blocks or log segments found. Creating new log.
I1014 19:25:38.017236 25324 log.cc:525] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Log is configured to *not* fsync() on all Append() calls
I1014 19:25:38.024127 25324 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: No bootstrap required, opened a new log
I1014 19:25:38.034338 25324 raft_consensus.cc:325] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER }
I1014 19:25:38.034651 25324 raft_consensus.cc:351] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:38.034730 25324 raft_consensus.cc:630] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000641d996c4cdba3b4481a74eda150, State: Initialized, Role: FOLLOWER
I1014 19:25:38.035478 25324 consensus_queue.cc:228] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER }
I1014 19:25:38.035696 25324 raft_consensus.cc:365] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I1014 19:25:38.035800 25324 raft_consensus.cc:457] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I1014 19:25:38.035898 25324 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Snoozing failure detection for 1.520s (starting election)
I1014 19:25:38.036025 25324 raft_consensus.cc:2795] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:38.038717 25324 raft_consensus.cc:479] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER }
I1014 19:25:38.039275 25324 leader_election.cc:278] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:38.044947 25347 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Snoozing failure detection for 2.111s (election complete)
I1014 19:25:38.045115 25347 raft_consensus.cc:2549] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 1 FOLLOWER]: Leader election won for term 1
I1014 19:25:38.046344 25347 raft_consensus.cc:595] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [term 1 LEADER]: Becoming Leader. State: Replica: f000641d996c4cdba3b4481a74eda150, State: Running, Role: LEADER
I1014 19:25:38.047082 25347 consensus_queue.cc:205] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER }
I1014 19:25:38.051246 25347 sys_catalog.cc:337] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f000641d996c4cdba3b4481a74eda150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER } }
I1014 19:25:38.051574 25352 sys_catalog.cc:337] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f000641d996c4cdba3b4481a74eda150. Latest consensus state: current_term: 1 leader_uuid: "f000641d996c4cdba3b4481a74eda150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER } }
I1014 19:25:38.051935 25347 sys_catalog.cc:340] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:38.051981 25352 sys_catalog.cc:340] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:38.055150 25324 sys_catalog.cc:337] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "f000641d996c4cdba3b4481a74eda150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000641d996c4cdba3b4481a74eda150" member_type: VOTER } }
I1014 19:25:38.055387 25324 sys_catalog.cc:340] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:38.058004 25324 sys_catalog.cc:424] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:38.060649 25118 master_main.cc:86] Master server successfully started.
W1014 19:25:38.061096 25362 fault_injection.cc:41] FAULT INJECTION ENABLED!
W1014 19:25:38.061189 25362 fault_injection.cc:42] THIS SERVER MAY CRASH!
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:38.198891 25330 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:38.199282 25330 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W1014 19:25:38.201750 25330 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:38.201892 25330 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.201.1
I1014 19:25:38.208622 25330 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.20.201.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.201.1
--webserver_port=0
--tserver_master_addrs=127.20.201.62:36022
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.201.1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
I1014 19:25:38.214826 25330 tablet_server_main.cc:85] Initializing tablet server...
I1014 19:25:38.216050 25330 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 421578us
I1014 19:25:38.216706 25330 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:38.216771 25330 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:38.217041 25330 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:38.217134 25330 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:38.219697 25330 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/instance:
uuid: "4320971132194d87a2a2c2d12e021054"
format_stamp: "Formatted at 2018-10-15 02:25:38 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:38.220329 25330 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal/instance:
uuid: "4320971132194d87a2a2c2d12e021054"
format_stamp: "Formatted at 2018-10-15 02:25:38 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:38.227430 25330 fs_manager.cc:503] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.001s
I1014 19:25:38.228554 25330 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:38.228996 25330 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:38.232633 25330 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I1014 19:25:38.232703 25330 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
uuid: "4320971132194d87a2a2c2d12e021054"
format_stamp: "Formatted at 2018-10-15 02:25:38 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:38.232841 25330 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:38.279590 25330 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:38.281626 25330 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:38.282697 25330 tablet_server_main.cc:90] Starting tablet server...
W1014 19:25:38.283368 25362 catalog_manager.cc:1084] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W1014 19:25:38.283491 25362 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I1014 19:25:38.283617 25359 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:38.287603 25359 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:38.433645 25330 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:33060
I1014 19:25:38.434274 25330 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:38.434355 25330 webserver.cc:186] Document root disabled
I1014 19:25:38.435310 25330 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:54240/
I1014 19:25:38.436120 25330 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.SingleMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
I1014 19:25:38.444953 25330 tablet_server_main.cc:93] Tablet server successfully started.
I1014 19:25:38.446274 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25330
I1014 19:25:38.471060 25574 heartbeater.cc:346] Connected to a master server at 127.20.201.62:36022
I1014 19:25:38.471290 25574 heartbeater.cc:426] Registering TS with master...
I1014 19:25:38.471885 25574 heartbeater.cc:475] Master 127.20.201.62:36022 requested a full tablet report, sending...
I1014 19:25:38.474323 25267 ts_manager.cc:98] Registered new tserver with Master: 4320971132194d87a2a2c2d12e021054 (127.20.201.1:33060)
I1014 19:25:38.476972 21284 external_mini_cluster.cc:535] 1 TS(s) registered with all masters
I1014 19:25:40.060114 25359 catalog_manager.cc:924] Generated new certificate authority record
W1014 19:25:40.061813 25359 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Initializing Kudu internal certificate authority: real 1.774s	user 0.024s	sys 0.001s
I1014 19:25:40.061883 25359 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:40.075918 25359 catalog_manager.cc:4062] T 00000000000000000000000000000000 P f000641d996c4cdba3b4481a74eda150: Generated new TSK 0
I1014 19:25:40.481119 25267 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.1:33477
I1014 19:25:41.483889 25574 heartbeater.cc:467] Master 127.20.201.62:36022 was elected leader, sending a full tablet report...
I1014 19:25:42.229321 25317 generic_service.cc:146] {username='jenkins'} at 127.20.201.62:57233 changed flags via RPC: catalog_manager_inject_latency_load_ca_info_ms from '2500' to '0'
I1014 19:25:43.500936 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25330
I1014 19:25:43.508839 25570 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:25:43.557134 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25330
I1014 19:25:43.566975 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 25118
W1014 19:25:43.567039 25239 connection.cc:512] server connection from 127.20.201.1:33477 recv error: Network error: failed to read from TLS socket (remote: 127.20.201.1:33477): Connection reset by peer (error 104)
I1014 19:25:43.567955 25317 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:25:43.671402 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 25118
[       OK ] SingleMasterConnectToClusterTest.ConnectToCluster (6051 ms)
[----------] 1 test from SingleMasterConnectToClusterTest (6051 ms total)

[----------] 1 test from MultiMasterConnectToClusterTest
[ RUN      ] MultiMasterConnectToClusterTest.ConnectToCluster
I1014 19:25:43.694669 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.62:56252
--webserver_interface=127.20.201.62
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--catalog_manager_inject_latency_load_ca_info_ms=2000
--raft_enable_pre_election=false
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1.0
--raft_heartbeat_interval_ms=100
--rpc_reuseport=true
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788 with env {}
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:43.914335 28481 flags.cc:406] Enabled unsafe flag: --catalog_manager_inject_latency_load_ca_info_ms=2000
W1014 19:25:43.914614 28481 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:43.914697 28481 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W1014 19:25:43.917337 28481 flags.cc:406] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=300
W1014 19:25:43.917402 28481 flags.cc:406] Enabled experimental flag: --raft_enable_pre_election=false
W1014 19:25:43.917513 28481 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W1014 19:25:43.917570 28481 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:43.917616 28481 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W1014 19:25:43.918076 28481 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I1014 19:25:43.930558 28481 master_main.cc:73] Master server non-default flags:
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1
--raft_enable_pre_election=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
--catalog_manager_inject_latency_load_ca_info_ms=2000
--ipki_ca_key_size=1024
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.62:56252
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.201.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
I1014 19:25:43.938374 28481 master_main.cc:80] Initializing master server...
I1014 19:25:43.939805 28481 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 424078us
I1014 19:25:43.941534 28481 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:43.941646 28481 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:43.942389 28481 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:43.942562 28481 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:43.946960 28481 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/instance:
uuid: "88e69d68a7914bf48d802a2f2e500b71"
format_stamp: "Formatted at 2018-10-15 02:25:43 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:43.947602 28481 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal/instance:
uuid: "88e69d68a7914bf48d802a2f2e500b71"
format_stamp: "Formatted at 2018-10-15 02:25:43 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:43.960655 28481 fs_manager.cc:503] Time spent creating directory manager: real 0.013s	user 0.004s	sys 0.008s
I1014 19:25:43.962882 28481 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:43.963512 28481 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:43.968946 28481 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I1014 19:25:43.969015 28481 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
uuid: "88e69d68a7914bf48d802a2f2e500b71"
format_stamp: "Formatted at 2018-10-15 02:25:43 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:43.969166 28481 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:44.009136 28481 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:44.010993 28481 master_main.cc:83] Starting Master server...
I1014 19:25:44.092559 28481 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.62:56252
I1014 19:25:44.093086 28481 webserver.cc:175] Starting webserver on 127.20.201.62:0
I1014 19:25:44.093130 28481 webserver.cc:186] Document root disabled
I1014 19:25:44.093993 28481 webserver.cc:313] Webserver started. Bound to: http://127.20.201.62:49703/
I1014 19:25:44.094602 28481 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-0/data/info.pb
I1014 19:25:44.098110 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 28481
I1014 19:25:44.098956 28663 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:44.098950 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.61:59248
--webserver_interface=127.20.201.61
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--catalog_manager_inject_latency_load_ca_info_ms=2000
--raft_enable_pre_election=false
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1.0
--raft_heartbeat_interval_ms=100
--rpc_reuseport=true
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788 with env {}
I1014 19:25:44.113265 28663 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.130661 28663 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:44.131516 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:59248: Network error: Client connection negotiation failed: client connection to 127.20.201.61:59248: connect: Connection refused (error 111)
I1014 19:25:44.150802 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } attempt: 1
W1014 19:25:44.151774 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:59248: Network error: Client connection negotiation failed: client connection to 127.20.201.61:59248: connect: Connection refused (error 111)
I1014 19:25:44.199995 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } attempt: 2
W1014 19:25:44.200976 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:59248: Network error: Client connection negotiation failed: client connection to 127.20.201.61:59248: connect: Connection refused (error 111)
I1014 19:25:44.312194 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } attempt: 3
W1014 19:25:44.313395 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:59248: Network error: Client connection negotiation failed: client connection to 127.20.201.61:59248: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:44.330265 28664 flags.cc:406] Enabled unsafe flag: --catalog_manager_inject_latency_load_ca_info_ms=2000
W1014 19:25:44.330646 28664 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:44.330762 28664 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W1014 19:25:44.333957 28664 flags.cc:406] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=300
W1014 19:25:44.334022 28664 flags.cc:406] Enabled experimental flag: --raft_enable_pre_election=false
W1014 19:25:44.334100 28664 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W1014 19:25:44.334156 28664 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:44.334204 28664 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W1014 19:25:44.334682 28664 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I1014 19:25:44.346189 28664 master_main.cc:73] Master server non-default flags:
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1
--raft_enable_pre_election=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal
--catalog_manager_inject_latency_load_ca_info_ms=2000
--ipki_ca_key_size=1024
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.61:59248
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.201.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
I1014 19:25:44.355041 28664 master_main.cc:80] Initializing master server...
I1014 19:25:44.355989 28664 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 424578us
I1014 19:25:44.357286 28664 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:44.357393 28664 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:44.358150 28664 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:44.358270 28664 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:44.362288 28664 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data/instance:
uuid: "cbc42c7d5db44e00883549e254bdd22e"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.363075 28664 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal/instance:
uuid: "cbc42c7d5db44e00883549e254bdd22e"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.376796 28664 fs_manager.cc:503] Time spent creating directory manager: real 0.013s	user 0.007s	sys 0.006s
I1014 19:25:44.379063 28664 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:44.379707 28664 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:44.384447 28664 fs_manager.cc:419] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I1014 19:25:44.384536 28664 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal
uuid: "cbc42c7d5db44e00883549e254bdd22e"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.384739 28664 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:44.471954 28664 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:44.473297 28664 master_main.cc:83] Starting Master server...
I1014 19:25:44.489528 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } attempt: 4
W1014 19:25:44.490351 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.61:59248: Network error: Client connection negotiation failed: client connection to 127.20.201.61:59248: connect: Connection refused (error 111)
I1014 19:25:44.512224 28664 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.61:59248
I1014 19:25:44.512725 28664 webserver.cc:175] Starting webserver on 127.20.201.61:0
I1014 19:25:44.512766 28664 webserver.cc:186] Document root disabled
I1014 19:25:44.513497 28664 webserver.cc:313] Webserver started. Bound to: http://127.20.201.61:37878/
I1014 19:25:44.514161 28664 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-1/data/info.pb
I1014 19:25:44.517076 28849 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:44.521762 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 28664
I1014 19:25:44.522639 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--ipki_ca_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.60:53788
--webserver_interface=127.20.201.60
--webserver_port=0
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--catalog_manager_inject_latency_load_ca_info_ms=2000
--raft_enable_pre_election=false
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1.0
--raft_heartbeat_interval_ms=100
--rpc_reuseport=true
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788 with env {}
I1014 19:25:44.526779 28849 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.538895 28849 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.559630 28849 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:44.560889 28849 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
I1014 19:25:44.580170 28849 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 1
W1014 19:25:44.581413 28849 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
I1014 19:25:44.629626 28849 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 2
W1014 19:25:44.630630 28849 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:44.726620 28851 flags.cc:406] Enabled unsafe flag: --catalog_manager_inject_latency_load_ca_info_ms=2000
W1014 19:25:44.726862 28851 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:44.726991 28851 flags.cc:406] Enabled unsafe flag: --never_fsync=true
W1014 19:25:44.729879 28851 flags.cc:406] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=300
W1014 19:25:44.729985 28851 flags.cc:406] Enabled experimental flag: --raft_enable_pre_election=false
W1014 19:25:44.730095 28851 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024
W1014 19:25:44.730188 28851 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:44.730260 28851 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512
W1014 19:25:44.730348 28851 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true
I1014 19:25:44.741935 28849 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 3
I1014 19:25:44.742658 28851 master_main.cc:73] Master server non-default flags:
--leader_failure_exp_backoff_max_delta_ms=300
--leader_failure_max_missed_heartbeat_periods=1
--raft_enable_pre_election=false
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal
--catalog_manager_inject_latency_load_ca_info_ms=2000
--ipki_ca_key_size=1024
--master_addresses=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788
--ipki_server_key_size=1024
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.201.60:53788
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.201.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
W1014 19:25:44.743028 28849 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
I1014 19:25:44.748992 28851 master_main.cc:80] Initializing master server...
I1014 19:25:44.749797 28851 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 424578us
I1014 19:25:44.750948 28851 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:44.751026 28851 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:44.751368 28851 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:44.751507 28851 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:44.754179 28851 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data/instance:
uuid: "4043e04130264237805f012d8e251bd4"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.754819 28851 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal/instance:
uuid: "4043e04130264237805f012d8e251bd4"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.767422 28851 fs_manager.cc:503] Time spent creating directory manager: real 0.012s	user 0.004s	sys 0.007s
I1014 19:25:44.769006 28851 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:44.769685 28851 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:44.774549 28851 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I1014 19:25:44.774638 28851 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal
uuid: "4043e04130264237805f012d8e251bd4"
format_stamp: "Formatted at 2018-10-15 02:25:44 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:44.774848 28851 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:44.793474 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } attempt: 5
I1014 19:25:44.799044 28663 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } has no permanent_uuid. Determining permanent_uuid...
W1014 19:25:44.800014 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
I1014 19:25:44.838238 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 1
W1014 19:25:44.839257 28663 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.20.201.60:53788: Network error: Client connection negotiation failed: client connection to 127.20.201.60:53788: connect: Connection refused (error 111)
I1014 19:25:44.845813 28851 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:44.846952 28851 master_main.cc:83] Starting Master server...
I1014 19:25:44.877045 28851 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.60:53788
I1014 19:25:44.877674 28851 webserver.cc:175] Starting webserver on 127.20.201.60:0
I1014 19:25:44.877718 28851 webserver.cc:186] Document root disabled
I1014 19:25:44.878494 28851 webserver.cc:313] Webserver started. Bound to: http://127.20.201.60:48126/
I1014 19:25:44.879108 28851 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/master-2/data/info.pb
I1014 19:25:44.881222 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 28851
I1014 19:25:44.882314 21284 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver
/data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data
--block_manager=log
--rpc_bind_addresses=127.20.201.1:0
--local_ip_for_outbound_sockets=127.20.201.1
--webserver_interface=127.20.201.1
--webserver_port=0
--tserver_master_addrs=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788
--never_fsync
--ipki_server_key_size=1024
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--logtostderr
--logbuflevel=-1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/logs
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags with env {}
I1014 19:25:44.882741 29034 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I1014 19:25:44.887460 28663 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 2
I1014 19:25:44.901058 29034 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.912307 28663 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Bootstrap starting.
I1014 19:25:44.915117 28663 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: No blocks or log segments found. Creating new log.
I1014 19:25:44.916224 28663 log.cc:525] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Log is configured to *not* fsync() on all Append() calls
I1014 19:25:44.916508 29034 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.919432 28849 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } attempt: 4
I1014 19:25:44.925176 28663 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: No bootstrap required, opened a new log
I1014 19:25:44.930080 29034 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } has no permanent_uuid. Determining permanent_uuid...
I1014 19:25:44.930143 28849 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Bootstrap starting.
I1014 19:25:44.933161 28849 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: No blocks or log segments found. Creating new log.
I1014 19:25:44.934201 28849 log.cc:525] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Log is configured to *not* fsync() on all Append() calls
I1014 19:25:44.936887 28663 raft_consensus.cc:325] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.937296 28663 raft_consensus.cc:351] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:44.937400 28663 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Initialized, Role: FOLLOWER
I1014 19:25:44.938069 28663 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.939079 29034 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Bootstrap starting.
I1014 19:25:44.943312 29034 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: No blocks or log segments found. Creating new log.
I1014 19:25:44.943542 28663 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.943841 28663 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:44.944305 29034 log.cc:525] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Log is configured to *not* fsync() on all Append() calls
I1014 19:25:44.945168 28663 sys_catalog.cc:424] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:44.945420 29043 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.945755 29043 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:44.946094 28849 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: No bootstrap required, opened a new log
I1014 19:25:44.947556 29043 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:44.949208 28481 master_main.cc:86] Master server successfully started.
I1014 19:25:44.949261 29043 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:25:44.949363 29043 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:44.951938 29043 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.953011 29043 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 1 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:44.953588 29043 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 1 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:44.954190 29034 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: No bootstrap required, opened a new log
I1014 19:25:44.956140 28849 raft_consensus.cc:325] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.956602 28849 raft_consensus.cc:351] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:44.956791 28849 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Initialized, Role: FOLLOWER
W1014 19:25:44.957074 29046 fault_injection.cc:41] FAULT INJECTION ENABLED!
W1014 19:25:44.957139 29046 fault_injection.cc:42] THIS SERVER MAY CRASH!
I1014 19:25:44.957643 28849 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.967478 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.967999 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:44.967656 28849 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.968467 28849 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:44.969604 29034 raft_consensus.cc:325] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.969926 29034 raft_consensus.cc:351] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I1014 19:25:44.970013 29034 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Initialized, Role: FOLLOWER
I1014 19:25:44.970647 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:44.970582 29034 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.971210 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:44.971369 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (vote granted)
I1014 19:25:44.973654 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 1.
W1014 19:25:44.974274 28805 tablet.cc:1720] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I1014 19:25:44.974325 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 1 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:44.974440 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won.
I1014 19:25:44.974519 28849 sys_catalog.cc:424] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:44.974923 29043 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:25:44.975008 29043 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 FOLLOWER]: Leader election won for term 1
I1014 19:25:44.977488 29043 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:44.977922 29043 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:44.978464 29034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.978760 29034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:44.980100 28664 master_main.cc:86] Master server successfully started.
I1014 19:25:44.979099 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.980741 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:25:44.981143 29051 fault_injection.cc:41] FAULT INJECTION ENABLED!
W1014 19:25:44.981210 29051 fault_injection.cc:42] THIS SERVER MAY CRASH!
I1014 19:25:44.981058 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:44.981251 29034 sys_catalog.cc:424] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: configured and running, proceeding with master startup.
I1014 19:25:44.981395 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 0 FOLLOWER]: Advancing to term 1
I1014 19:25:44.981765 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (vote granted)
I1014 19:25:44.984642 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 1.
I1014 19:25:44.985169 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 1 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:44.985764 29052 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:44.986017 29052 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:44.989078 29054 catalog_manager.cc:525] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I1014 19:25:44.989593 28851 master_main.cc:86] Master server successfully started.
W1014 19:25:45.012501 28617 tablet.cc:1720] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I1014 19:25:45.092897 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 1 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I1014 19:25:45.093369 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch)
I1014 19:25:45.094637 29052 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I1014 19:25:45.094914 29043 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W1014 19:25:45.097836 28990 tablet.cc:1720] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I1014 19:25:45.099086 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.099324 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.106271 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.106501 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1014 19:25:45.113415 29036 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W1014 19:25:45.114013 29036 flags.cc:406] Enabled unsafe flag: --never_fsync=true
I1014 19:25:45.114305 29043 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.114545 29043 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.114676 29043 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.114840 29043 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:25:45.117568 29036 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024
W1014 19:25:45.117761 29036 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.201.1
I1014 19:25:45.118731 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.118949 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.121872 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.122047 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.139034 29036 tablet_server_main.cc:78] Tablet server non-default flags:
--fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data
--fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
--ipki_server_key_size=1024
--rpc_bind_addresses=127.20.201.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.201.1
--webserver_port=0
--tserver_master_addrs=127.20.201.62:56252,127.20.201.61:59248,127.20.201.60:53788
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.201.1
--log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.9.0-SNAPSHOT
revision 5894af6ff12291ffab2d3304b34f77fe6d112771
build type FASTDEBUG
built by jenkins at 14 Oct 2018 18:30:25 PST on quasar-ixxjpd-1.vpc.cloudera.com
build id 163349
ASAN enabled
I1014 19:25:45.145718 29036 tablet_server_main.cc:85] Initializing tablet server...
I1014 19:25:45.147239 29036 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 425078us
I1014 19:25:45.148121 29036 fs_manager.cc:263] Metadata directory not provided
I1014 19:25:45.148229 29036 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory
I1014 19:25:45.148630 29036 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file
I1014 19:25:45.148744 29036 server_base.cc:433] Attempting to create new FS layout instead
I1014 19:25:45.152173 29036 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/instance:
uuid: "0e3c7f0d1cb242518cabae35a1c0dabd"
format_stamp: "Formatted at 2018-10-15 02:25:45 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:45.152901 29036 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal/instance:
uuid: "0e3c7f0d1cb242518cabae35a1c0dabd"
format_stamp: "Formatted at 2018-10-15 02:25:45 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:45.166525 29036 fs_manager.cc:503] Time spent creating directory manager: real 0.013s	user 0.006s	sys 0.005s
I1014 19:25:45.167958 29036 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go
I1014 19:25:45.168607 29036 file_cache.cc:466] Constructed file cache lbm with capacity 13107
I1014 19:25:45.172956 29036 fs_manager.cc:419] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I1014 19:25:45.173060 29036 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
uuid: "0e3c7f0d1cb242518cabae35a1c0dabd"
format_stamp: "Formatted at 2018-10-15 02:25:45 on quasar-ixxjpd-1.vpc.cloudera.com"
I1014 19:25:45.173269 29036 fs_report.cc:352] FS layout report
--------------------
wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/wal
1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I1014 19:25:45.203835 29036 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I1014 19:25:45.205590 29036 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets)
I1014 19:25:45.206039 29036 tablet_server_main.cc:90] Starting tablet server...
I1014 19:25:45.208333 29050 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:45.208453 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (starting election)
I1014 19:25:45.208536 29050 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:45.210808 29050 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.211802 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 2 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.212430 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 2 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:45.212991 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:45.213316 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 2 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.213798 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:45.213999 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 2 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.213958 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 2 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 2 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.214301 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 2 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 2 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.214382 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost.
I1014 19:25:45.214929 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (election complete)
I1014 19:25:45.215029 29050 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I1014 19:25:45.229964 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 1. Current term is 2. Ops: []
I1014 19:25:45.230518 29052 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I1014 19:25:45.230772 29052 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 LEADER]: Stepping down as leader of term 1
I1014 19:25:45.230823 29052 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:45.231107 29052 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.231329 29052 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:45.231432 29052 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:45.231637 29052 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:45.282143 29049 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:45.282259 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (starting election)
I1014 19:25:45.282326 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 1 FOLLOWER]: Advancing to term 2
I1014 19:25:45.284651 29049 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.285745 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 2 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.286303 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 2 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.286782 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:45.286993 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (vote granted)
I1014 19:25:45.289108 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 2.
I1014 19:25:45.289568 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 2 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.289695 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won.
I1014 19:25:45.290016 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:25:45.290118 29049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 FOLLOWER]: Leader election won for term 2
I1014 19:25:45.290115 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:45.290364 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 2: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:45.290863 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 2 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 2: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:45.299309 29049 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:45.299739 29049 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.301564 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 2 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.301857 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.305917 29036 rpc_server.cc:205] RPC server started. Bound to: 127.20.201.1:57994
I1014 19:25:45.306375 29036 webserver.cc:175] Starting webserver on 127.20.201.1:0
I1014 19:25:45.306413 29036 webserver.cc:186] Document root disabled
I1014 19:25:45.307118 29036 webserver.cc:313] Webserver started. Bound to: http://127.20.201.1:57600/
I1014 19:25:45.307632 29036 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/master_cert_authority-itest.0.MultiMasterConnectToClusterTest.ConnectToCluster.1539570329182132-21284/minicluster-data/ts-0/data/info.pb
I1014 19:25:45.316939 21284 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29036
I1014 19:25:45.319520 29036 tablet_server_main.cc:93] Tablet server successfully started.
I1014 19:25:45.323279 29050 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:45.323372 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.204s (starting election)
I1014 19:25:45.323475 29050 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 2 FOLLOWER]: Advancing to term 3
I1014 19:25:45.326472 29050 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.327278 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 3 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.328203 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 3 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:45.328264 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:45.328485 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 2 FOLLOWER]: Advancing to term 3
I1014 19:25:45.328569 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (vote granted)
I1014 19:25:45.331025 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 3.
I1014 19:25:45.331248 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:45.331486 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 3 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.332043 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 3 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.332129 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won.
I1014 19:25:45.333652 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.168s (election complete)
I1014 19:25:45.333779 29050 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 FOLLOWER]: Leader election won for term 3
I1014 19:25:45.334097 29050 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:45.334637 29050 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.336971 29303 heartbeater.cc:346] Connected to a master server at 127.20.201.61:59248
I1014 19:25:45.337380 29303 heartbeater.cc:426] Registering TS with master...
I1014 19:25:45.337805 29302 heartbeater.cc:346] Connected to a master server at 127.20.201.62:56252
I1014 19:25:45.337883 29302 heartbeater.cc:426] Registering TS with master...
I1014 19:25:45.337954 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 3 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 3 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.338351 29304 heartbeater.cc:346] Connected to a master server at 127.20.201.60:53788
I1014 19:25:45.338521 29304 heartbeater.cc:426] Registering TS with master...
I1014 19:25:45.339165 29302 heartbeater.cc:475] Master 127.20.201.62:56252 requested a full tablet report, sending...
I1014 19:25:45.339308 29304 heartbeater.cc:475] Master 127.20.201.60:53788 requested a full tablet report, sending...
I1014 19:25:45.339732 29303 heartbeater.cc:475] Master 127.20.201.61:59248 requested a full tablet report, sending...
I1014 19:25:45.340394 29314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.340957 29314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.341296 28816 ts_manager.cc:98] Registered new tserver with Master: 0e3c7f0d1cb242518cabae35a1c0dabd (127.20.201.1:57994)
I1014 19:25:45.341518 29001 ts_manager.cc:98] Registered new tserver with Master: 0e3c7f0d1cb242518cabae35a1c0dabd (127.20.201.1:57994)
I1014 19:25:45.342022 28630 ts_manager.cc:98] Registered new tserver with Master: 0e3c7f0d1cb242518cabae35a1c0dabd (127.20.201.1:57994)
I1014 19:25:45.347000 29316 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:45.350642 21284 external_mini_cluster.cc:535] 1 TS(s) registered with all masters
I1014 19:25:45.352509 29316 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:25:45.352689 29316 fault_injection.cc:41] FAULT INJECTION ENABLED!
W1014 19:25:45.352751 29316 fault_injection.cc:42] THIS SERVER MAY CRASH!
I1014 19:25:45.419602 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 3 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 2. Current term is 3. Ops: []
I1014 19:25:45.420331 29290 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I1014 19:25:45.420686 29049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 LEADER]: Stepping down as leader of term 2
I1014 19:25:45.420744 29049 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:45.420998 29049 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.421164 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:45.421241 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:45.421401 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 2 FOLLOWER]: Advancing to term 3
I1014 19:25:45.428045 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 3 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I1014 19:25:45.428663 29314 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I1014 19:25:45.430248 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.430759 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.433286 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.433511 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.433508 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.433733 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.434288 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I1014 19:25:45.434362 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 1
I1014 19:25:45.434406 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I1014 19:25:45.434537 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:45.435642 29050 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I1014 19:25:45.439512 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.440008 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.439652 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.440426 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.498594 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.498854 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.508836 29316 catalog_manager.cc:924] Generated new certificate authority record
I1014 19:25:45.510807 29316 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:45.528036 29316 catalog_manager.cc:4062] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Generated new TSK 0
I1014 19:25:45.633093 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:45.633190 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:45.633262 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 3 FOLLOWER]: Advancing to term 4
I1014 19:25:45.635354 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.635885 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 4 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:45.636076 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 4 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.636443 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:45.636669 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 4 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.636863 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:45.637110 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 4 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.637715 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 4 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 4 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.637882 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 4 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 4 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.637964 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost.
I1014 19:25:45.638217 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:25:45.638288 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I1014 19:25:45.648175 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 3. Current term is 4. Ops: []
I1014 19:25:45.648881 29050 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I1014 19:25:45.649241 29050 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 LEADER]: Stepping down as leader of term 3
I1014 19:25:45.649304 29050 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:45.649544 29050 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.649760 29050 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:45.649842 29050 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:45.650051 29050 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 3 FOLLOWER]: Advancing to term 4
I1014 19:25:45.709151 29049 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:45.709271 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:25:45.709367 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 3 FOLLOWER]: Advancing to term 4
I1014 19:25:45.711853 29049 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.712385 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 4 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.713330 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:45.713568 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 4: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:45.713987 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 4 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 4: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:45.714138 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 4 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.714826 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:45.715059 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:25:45.717873 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 4.
I1014 19:25:45.719463 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 4 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.719535 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won.
I1014 19:25:45.719765 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (election complete)
I1014 19:25:45.719835 29049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 FOLLOWER]: Leader election won for term 4
I1014 19:25:45.719986 29049 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:45.720158 29049 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.721055 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 4 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.721249 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.777377 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:45.777462 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.189s (starting election)
I1014 19:25:45.777559 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 4 FOLLOWER]: Advancing to term 5
I1014 19:25:45.779836 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.780408 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 5 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:45.780611 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 5 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.781239 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:45.781446 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 5 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.781498 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:45.781672 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 4 FOLLOWER]: Advancing to term 5
I1014 19:25:45.781848 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 5 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 5 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.789551 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:25:45.796331 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 5.
I1014 19:25:45.797835 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 5 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:45.797931 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won.
I1014 19:25:45.798204 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.210s (election complete)
I1014 19:25:45.798280 29177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 FOLLOWER]: Leader election won for term 5
I1014 19:25:45.798712 29177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:45.798977 29177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.800091 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.800410 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.833184 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 4. Current term is 5. Ops: []
I1014 19:25:45.833732 29049 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I1014 19:25:45.833938 29049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I1014 19:25:45.834039 29049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 LEADER]: Stepping down as leader of term 4
I1014 19:25:45.834084 29049 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:45.834261 29049 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.834450 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:45.834529 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:45.834641 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 4 FOLLOWER]: Advancing to term 5
I1014 19:25:45.849493 29051 catalog_manager.cc:1082] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: acquiring CA information for follower catalog manager: success
I1014 19:25:45.854030 29051 catalog_manager.cc:1110] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:45.854305 29298 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:45.856875 29298 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:45.884661 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I1014 19:25:45.884768 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 5
I1014 19:25:45.884836 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I1014 19:25:45.884928 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:45.885562 29494 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I1014 19:25:45.886914 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.887228 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.888161 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.888373 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:45.888836 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 5 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I1014 19:25:45.889382 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I1014 19:25:45.890100 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.890352 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.891216 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.891686 29372 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:45.892307 29372 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.892401 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:45.991700 29050 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:45.991798 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:25:45.992043 29050 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 5 FOLLOWER]: Advancing to term 6
I1014 19:25:45.992640 29054 catalog_manager.cc:1110] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:45.994310 29050 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:45.994804 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 6 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:45.995364 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:45.995553 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 6 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.996002 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 6 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 6 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.996191 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 6 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:45.997485 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:45.997701 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 6 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.997982 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 6 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 6 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:45.998057 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost.
I1014 19:25:45.998356 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:25:45.998446 29050 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I1014 19:25:46.003262 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.003463 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.004191 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 5. Current term is 6. Ops: []
I1014 19:25:46.008824 29494 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I1014 19:25:46.009048 29494 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Stepping down as leader of term 5
I1014 19:25:46.009099 29494 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:46.009323 29494 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.009553 29494 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:46.009649 29494 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:46.009826 29494 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 5 FOLLOWER]: Advancing to term 6
I1014 19:25:46.090008 29290 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:46.090215 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:25:46.090327 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 5 FOLLOWER]: Advancing to term 6
I1014 19:25:46.093560 29290 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.094141 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 6 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.094768 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.094930 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (vote granted)
I1014 19:25:46.097065 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 6.
I1014 19:25:46.097429 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 6 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.097594 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won.
I1014 19:25:46.097842 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:25:46.097954 29049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 FOLLOWER]: Leader election won for term 6
I1014 19:25:46.099175 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 6 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.099892 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.100108 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 6: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:46.100539 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 6 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 6: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:46.105407 29049 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:46.105753 29049 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.107022 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 6 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.107365 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.116704 29050 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:46.116791 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.198s (starting election)
I1014 19:25:46.116874 29050 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 6 FOLLOWER]: Advancing to term 7
I1014 19:25:46.118997 29050 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.119498 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 7 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.119702 29050 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 7 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.120148 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.120306 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 6 FOLLOWER]: Advancing to term 7
I1014 19:25:46.120379 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (vote granted)
I1014 19:25:46.122519 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 7.
I1014 19:25:46.122839 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 7 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.122921 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won.
I1014 19:25:46.123073 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:46.123345 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 7 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.123826 29050 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:25:46.123821 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 7 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 7 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.124240 29050 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 FOLLOWER]: Leader election won for term 7
I1014 19:25:46.124464 29050 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:46.124662 29050 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.125809 29372 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.126020 29372 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.126564 29579 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:46.128823 29579 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:46.191074 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 6. Current term is 7. Ops: []
I1014 19:25:46.191845 29290 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I1014 19:25:46.192122 29290 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I1014 19:25:46.192232 29049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 LEADER]: Stepping down as leader of term 6
I1014 19:25:46.192287 29049 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:46.192565 29049 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.192804 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:46.192867 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:46.192987 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 6 FOLLOWER]: Advancing to term 7
I1014 19:25:46.226461 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I1014 19:25:46.227074 29050 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I1014 19:25:46.228570 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.228772 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.229338 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 7 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I1014 19:25:46.229461 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 6
I1014 19:25:46.229548 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 6.7
I1014 19:25:46.229689 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:46.230597 29050 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I1014 19:25:46.231942 29050 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.232228 29050 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.233147 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.233513 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.233501 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.233798 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.234609 29372 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.234896 29372 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.235417 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.235625 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.336313 29049 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:46.336397 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:46.336501 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 7 FOLLOWER]: Advancing to term 8
I1014 19:25:46.337986 29049 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.338434 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 8 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.338626 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 8 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.339082 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.339207 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.339294 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 8 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.339411 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 8 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.339742 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 8 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 8 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.339865 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 8 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 8 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.339937 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost.
I1014 19:25:46.340167 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:25:46.340242 29049 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I1014 19:25:46.356228 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 7. Current term is 8. Ops: []
I1014 19:25:46.356770 29372 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I1014 19:25:46.356995 29621 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Stepping down as leader of term 7
I1014 19:25:46.357064 29621 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:46.357319 29621 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.357540 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:46.357614 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:46.357770 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 7 FOLLOWER]: Advancing to term 8
I1014 19:25:46.410967 29494 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:46.411058 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:25:46.411175 29494 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 7 FOLLOWER]: Advancing to term 8
I1014 19:25:46.413429 29494 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.413980 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 8 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.414206 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 8 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.414830 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.415033 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (vote granted)
I1014 19:25:46.415412 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:46.415643 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 8: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:46.416334 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 8 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 8: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:46.417183 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 8.
I1014 19:25:46.417482 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 8 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.417551 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won.
I1014 19:25:46.417745 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:25:46.417799 29494 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 FOLLOWER]: Leader election won for term 8
I1014 19:25:46.417928 29494 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:46.418118 29494 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.419199 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 8 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.419405 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.443997 29049 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:46.444093 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:46.444173 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 8 FOLLOWER]: Advancing to term 9
I1014 19:25:46.446542 29049 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.447158 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 9 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.447870 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 9 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.447845 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.448025 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 9 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.448443 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 9 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 9 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.448428 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.448611 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 8 FOLLOWER]: Advancing to term 9
I1014 19:25:46.448712 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:25:46.450314 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 9.
I1014 19:25:46.450587 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 9 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.450649 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won.
I1014 19:25:46.450863 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.221s (election complete)
I1014 19:25:46.450939 29049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 FOLLOWER]: Leader election won for term 9
I1014 19:25:46.451086 29049 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:46.451256 29049 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.452296 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.452607 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.509605 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 9 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 8. Current term is 9. Ops: []
I1014 19:25:46.510254 29177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I1014 19:25:46.510565 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 LEADER]: Stepping down as leader of term 8
I1014 19:25:46.510619 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:46.510855 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.511050 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:46.511126 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:46.511242 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 8 FOLLOWER]: Advancing to term 9
I1014 19:25:46.529847 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 9 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I1014 19:25:46.530584 29290 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I1014 19:25:46.531991 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.532259 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.533354 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.533596 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.534339 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 9 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I1014 19:25:46.534401 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 7
I1014 19:25:46.534447 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 8.8
I1014 19:25:46.534556 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:46.535089 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.535204 29049 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I1014 19:25:46.535318 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.536701 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.536923 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.537052 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.537223 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.634995 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:46.635093 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:46.635179 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 9 FOLLOWER]: Advancing to term 10
I1014 19:25:46.637059 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.637308 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:46.637385 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:46.637399 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 9. Current term is 10. Ops: []
I1014 19:25:46.637481 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 9 FOLLOWER]: Advancing to term 10
I1014 19:25:46.637639 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 10 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.638034 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 10 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.638078 29049 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I1014 19:25:46.638435 29290 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 LEADER]: Stepping down as leader of term 9
I1014 19:25:46.638557 29290 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:46.638741 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:46.638793 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.638926 29290 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.639137 29290 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:46.639204 29290 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:46.639349 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 9 FOLLOWER]: Advancing to term 10
I1014 19:25:46.640429 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.640730 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 10: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:46.641216 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 10 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 10: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:46.641358 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 10 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.641676 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (vote granted)
I1014 19:25:46.641690 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 10 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.642014 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:46.642251 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.642392 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 10: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:46.642675 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 10 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 10: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:46.643975 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 10.
I1014 19:25:46.644129 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 10 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:46.644584 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 10 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 10 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:46.644662 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost.
I1014 19:25:46.644876 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:25:46.645256 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I1014 19:25:46.645901 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 10 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.645975 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won.
I1014 19:25:46.646250 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:25:46.646358 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Leader election won for term 10
I1014 19:25:46.646706 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:46.646944 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.648183 29372 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.648507 29372 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.745817 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I1014 19:25:46.746300 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I1014 19:25:46.747527 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.747820 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.749083 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.749428 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.749794 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I1014 19:25:46.749933 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.750157 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.750329 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I1014 19:25:46.751768 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.752028 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.752157 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.752334 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:46.827683 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.827932 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.849831 29290 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:46.849913 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (starting election)
I1014 19:25:46.849992 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 10 FOLLOWER]: Advancing to term 11
I1014 19:25:46.851800 29290 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.852320 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 11 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.852493 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 11 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.853222 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.853292 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.853451 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 11 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.853512 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 11 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.853909 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 11 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 11 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.854022 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 11 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 11 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.854115 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost.
I1014 19:25:46.854329 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:25:46.854403 29290 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I1014 19:25:46.861724 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 10. Current term is 11. Ops: []
I1014 19:25:46.862212 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I1014 19:25:46.862545 29813 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 LEADER]: Stepping down as leader of term 10
I1014 19:25:46.862594 29813 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:46.862817 29813 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.862960 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:46.863015 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:46.863122 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 10 FOLLOWER]: Advancing to term 11
I1014 19:25:46.895982 29046 catalog_manager.cc:1082] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: acquiring CA information for follower catalog manager: success
I1014 19:25:46.898818 29046 catalog_manager.cc:1110] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I1014 19:25:46.899106 29055 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:46.902268 29055 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:46.929409 29494 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:46.929553 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:25:46.929672 29494 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 10 FOLLOWER]: Advancing to term 11
I1014 19:25:46.932910 29494 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.933528 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 11 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:46.933792 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 11 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.934296 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:46.934536 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 11: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:46.934998 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 11 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 11: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:46.935500 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.935649 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:25:46.937119 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 11.
I1014 19:25:46.937438 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 11 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.937491 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won.
I1014 19:25:46.937705 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:25:46.937881 29494 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 FOLLOWER]: Leader election won for term 11
I1014 19:25:46.938166 29494 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:46.938455 29494 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.939965 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 11 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.940254 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:46.962775 29290 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:46.962859 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (starting election)
I1014 19:25:46.962924 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 11 FOLLOWER]: Advancing to term 12
I1014 19:25:46.965219 29290 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.965911 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 12 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:46.966274 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 12 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.966634 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:46.966836 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 12 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.967247 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:46.967452 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 11 FOLLOWER]: Advancing to term 12
I1014 19:25:46.967553 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (vote granted)
I1014 19:25:46.969858 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 12.
I1014 19:25:46.970238 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 12 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:46.970312 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won.
I1014 19:25:46.970916 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 12 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 12 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:46.971057 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.215s (election complete)
I1014 19:25:46.971153 29290 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 FOLLOWER]: Leader election won for term 12
I1014 19:25:46.971251 29290 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:46.971431 29290 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:46.972489 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:46.972689 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.025807 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 11. Current term is 12. Ops: []
I1014 19:25:47.026417 29494 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I1014 19:25:47.026746 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 LEADER]: Stepping down as leader of term 11
I1014 19:25:47.026839 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:47.027078 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.027364 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:47.027454 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:47.027647 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 11 FOLLOWER]: Advancing to term 12
I1014 19:25:47.059265 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I1014 19:25:47.059924 29049 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I1014 19:25:47.061184 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.061532 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.062458 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.062726 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.063014 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 12 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I1014 19:25:47.063086 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 9
I1014 19:25:47.063120 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 11.10
I1014 19:25:47.063179 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:47.063554 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.063769 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.063890 29290 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I1014 19:25:47.065217 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.065598 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.065744 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.065933 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.166136 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:47.166232 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:25:47.166301 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 12 FOLLOWER]: Advancing to term 13
I1014 19:25:47.169385 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.170115 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 13 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.170397 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 13 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.170886 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.171108 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 13 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.171103 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.171306 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 13 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.171607 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 13 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 13 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.171841 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 13 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 13 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.171972 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost.
I1014 19:25:47.172796 29290 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.173055 29290 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.172505 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:25:47.174515 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I1014 19:25:47.174655 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 12. Current term is 13. Ops: []
I1014 19:25:47.175331 29290 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I1014 19:25:47.176772 29049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 LEADER]: Stepping down as leader of term 12
I1014 19:25:47.177211 29049 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:47.177574 29049 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.177865 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:47.177944 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:47.178118 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 12 FOLLOWER]: Advancing to term 13
I1014 19:25:47.262363 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:47.262495 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:25:47.262588 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 12 FOLLOWER]: Advancing to term 13
I1014 19:25:47.264266 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.264847 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 13 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.265056 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 13 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.265744 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.265926 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (vote granted)
I1014 19:25:47.266888 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:47.267170 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 13: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:47.267551 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 13 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 13: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:47.268226 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 13.
I1014 19:25:47.268549 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 13 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.268613 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won.
I1014 19:25:47.268853 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:25:47.268930 29813 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 FOLLOWER]: Leader election won for term 13
I1014 19:25:47.269080 29813 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:47.269284 29813 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.270563 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 13 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.270727 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.320612 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:47.320706 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.160s (starting election)
I1014 19:25:47.320791 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 13 FOLLOWER]: Advancing to term 14
I1014 19:25:47.322716 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.323232 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 14 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.323487 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 14 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.324008 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.324156 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 13 FOLLOWER]: Advancing to term 14
I1014 19:25:47.324143 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.324235 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:25:47.324368 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 14 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.324774 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 14 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 14 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.325773 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 14.
I1014 19:25:47.326061 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 14 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.326138 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won.
I1014 19:25:47.326407 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.176s (election complete)
I1014 19:25:47.326750 29177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 FOLLOWER]: Leader election won for term 14
I1014 19:25:47.326886 29177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:47.327082 29177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.328027 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.328256 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.360249 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 14 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 13. Current term is 14. Ops: []
I1014 19:25:47.360759 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I1014 19:25:47.361013 29813 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 LEADER]: Stepping down as leader of term 13
I1014 19:25:47.361068 29813 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:47.361289 29813 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.361439 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:47.361534 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:47.361662 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 13 FOLLOWER]: Advancing to term 14
I1014 19:25:47.422742 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 14 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I1014 19:25:47.423257 29494 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I1014 19:25:47.424868 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.425155 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.425894 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.426115 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.426887 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 14 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I1014 19:25:47.426962 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 10
I1014 19:25:47.427026 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 13.11
I1014 19:25:47.427125 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:47.427791 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I1014 19:25:47.429179 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.429396 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.429565 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.429757 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.430626 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 14 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.430816 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.527154 29049 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:47.527257 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:25:47.527333 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 14 FOLLOWER]: Advancing to term 15
I1014 19:25:47.529464 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:47.529579 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (starting election)
I1014 19:25:47.529487 29049 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.529666 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 14 FOLLOWER]: Advancing to term 15
I1014 19:25:47.529857 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 14. Current term is 15. Ops: []
I1014 19:25:47.530009 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 15 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.530198 29049 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 15 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.530683 29177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I1014 19:25:47.530995 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 LEADER]: Stepping down as leader of term 14
I1014 19:25:47.531092 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:47.531190 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:47.531332 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.531540 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:47.531611 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:47.531544 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.531746 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 14 FOLLOWER]: Advancing to term 15
I1014 19:25:47.532176 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.532651 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 15 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.532835 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 15 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.533082 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 15: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:47.533366 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:47.533648 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.533866 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 15 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 15: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:47.533866 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 15: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:47.534376 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (vote granted)
I1014 19:25:47.534754 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 15 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 15: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:47.535781 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 15.
I1014 19:25:47.535969 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 15 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:47.536165 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 15 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.536226 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won.
I1014 19:25:47.536420 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 15 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 15 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:47.536459 29049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (election complete)
I1014 19:25:47.536538 29049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Leader election won for term 15
I1014 19:25:47.536556 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost.
I1014 19:25:47.536684 29049 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:47.536859 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:25:47.536973 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I1014 19:25:47.536893 29049 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.537830 29997 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.538110 29997 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.635440 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I1014 19:25:47.636018 29997 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I1014 19:25:47.637091 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.637284 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.638617 29997 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.638813 29997 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.638665 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.638850 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:47.638885 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.638923 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.215s (starting election)
I1014 19:25:47.638989 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 15 FOLLOWER]: Advancing to term 16
I1014 19:25:47.641255 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.641602 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 16 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 15. Current term is 16. Ops: []
I1014 19:25:47.641862 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 16 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.642069 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 16 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.642204 29049 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I1014 19:25:47.642459 29290 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 LEADER]: Stepping down as leader of term 15
I1014 19:25:47.642560 29290 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:47.642608 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:47.642650 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.642784 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 16 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.642832 29290 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.643054 29290 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:47.643069 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 16 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 16 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.643107 29290 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:47.643195 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 15 FOLLOWER]: Advancing to term 16
I1014 19:25:47.645005 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I1014 19:25:47.645364 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 16 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I1014 19:25:47.645442 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost.
I1014 19:25:47.645689 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.213s (election complete)
I1014 19:25:47.645758 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I1014 19:25:47.664266 29055 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:47.665295 29055 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:47.666973 29055 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 1 vs 15: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.667259 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.667487 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.667695 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.667889 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.668062 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.668246 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.668437 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.668618 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.668790 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 15 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.709048 29298 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.852s	user 0.002s	sys 0.000s
I1014 19:25:47.709151 29298 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:47.710098 29298 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:47.711381 29298 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 2 vs 16: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.711632 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.711789 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.711941 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.712177 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.712445 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.712740 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.712929 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.713090 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.713306 29298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.720243 29290 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:47.720309 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:47.720368 29290 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 16 FOLLOWER]: Advancing to term 17
I1014 19:25:47.721796 29290 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.722151 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 17 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:47.722324 29290 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 17 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.722858 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:47.722930 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.723065 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 17 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.723081 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 16 FOLLOWER]: Advancing to term 17
I1014 19:25:47.723155 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (vote granted)
I1014 19:25:47.723439 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 17 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 17 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.724921 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 17.
I1014 19:25:47.728499 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 17 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.728566 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won.
I1014 19:25:47.728983 29290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (election complete)
I1014 19:25:47.729059 29290 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 FOLLOWER]: Leader election won for term 17
I1014 19:25:47.729208 29290 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:47.729424 29290 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.730460 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.730671 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.730849 29298 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:47.732841 29298 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:47.738649 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:47.738750 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:47.738831 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 15 FOLLOWER]: Advancing to term 16
I1014 19:25:47.740842 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.741341 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 16 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.742066 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.742230 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 16 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.742436 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 16 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
W1014 19:25:47.742729 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 16 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 16 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.742779 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I1014 19:25:47.743000 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:25:47.743127 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 16 FOLLOWER]: Advancing to term 17
I1014 19:25:47.743477 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.743700 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 16. Current term is 17.
W1014 19:25:47.744336 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 16 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 16. Current term is 17.
I1014 19:25:47.744966 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 16 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.819696 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I1014 19:25:47.820201 29049 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I1014 19:25:47.822015 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.822010 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.822342 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.822260 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.822917 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.823112 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:47.823479 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 17 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I1014 19:25:47.824013 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.824074 29290 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I1014 19:25:47.824193 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.825315 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.825630 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.825528 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.825726 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:47.852671 29298 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:47.853647 29298 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:47.857545 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.857848 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:25:47.929913 29579 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.801s	user 0.002s	sys 0.000s
I1014 19:25:47.929968 29579 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:47.931071 29579 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:47.932826 29579 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 7 vs 17: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.933187 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.933532 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.933825 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.934142 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.934590 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:47.934875 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 17 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:47.960418 29494 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:47.960533 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (starting election)
I1014 19:25:47.960608 29494 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 17 FOLLOWER]: Advancing to term 18
I1014 19:25:47.962782 29494 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.963270 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 18 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:47.963481 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 18 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:47.963824 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:47.964010 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 18 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.964083 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:47.964258 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 18 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.964385 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 18 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 18 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.964695 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 18 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 18 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:47.964768 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost.
I1014 19:25:47.964994 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:25:47.965118 29494 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I1014 19:25:47.980588 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 17. Current term is 18. Ops: []
I1014 19:25:47.981124 30244 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I1014 19:25:47.981494 30244 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Stepping down as leader of term 17
I1014 19:25:47.981590 30244 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:47.981920 30244 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:47.982286 30244 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:47.982357 30244 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:47.982615 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 17 FOLLOWER]: Advancing to term 18
I1014 19:25:48.047590 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:48.047682 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (starting election)
I1014 19:25:48.047786 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 17 FOLLOWER]: Advancing to term 18
I1014 19:25:48.049644 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.050155 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 18 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.050318 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 18 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.051589 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.051620 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.051795 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 18: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:48.051775 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (vote granted)
I1014 19:25:48.052186 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 18 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 18: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:48.053392 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 18.
I1014 19:25:48.053732 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 18 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.053812 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won.
I1014 19:25:48.054076 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (election complete)
I1014 19:25:48.054150 29813 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 FOLLOWER]: Leader election won for term 18
I1014 19:25:48.054255 29813 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.054538 29813 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.055553 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 18 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.055702 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.055928 29579 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:48.057399 29579 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:48.067625 29494 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.067703 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.186s (starting election)
I1014 19:25:48.067777 29494 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 18 FOLLOWER]: Advancing to term 19
I1014 19:25:48.069375 29494 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.069808 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 19 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.069962 29494 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 19 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.070423 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.070565 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 18 FOLLOWER]: Advancing to term 19
I1014 19:25:48.070655 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (vote granted)
I1014 19:25:48.072131 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 19.
I1014 19:25:48.072451 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 19 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.072517 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won.
I1014 19:25:48.072743 29494 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.164s (election complete)
I1014 19:25:48.072816 29494 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 FOLLOWER]: Leader election won for term 19
I1014 19:25:48.072955 29494 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:48.073143 29494 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.074026 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.074221 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.074435 29055 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:48.074956 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.075153 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 19 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.075614 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 19 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 19 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.076238 29055 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:48.132829 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 18. Current term is 19. Ops: []
I1014 19:25:48.133432 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I1014 19:25:48.133620 29621 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I1014 19:25:48.133708 29813 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 LEADER]: Stepping down as leader of term 18
I1014 19:25:48.133767 29813 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.134006 29813 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.134147 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:48.134194 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.134281 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 18 FOLLOWER]: Advancing to term 19
I1014 19:25:48.170811 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 19 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I1014 19:25:48.170876 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 14
I1014 19:25:48.170920 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 18.15
I1014 19:25:48.170974 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:48.171485 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I1014 19:25:48.172902 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.173169 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.173867 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.174088 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.174446 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I1014 19:25:48.174867 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.175057 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.175037 30402 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I1014 19:25:48.176585 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.176874 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.177114 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.177271 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.264721 29494 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 19 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.264904 29494 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.274833 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:48.274915 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:25:48.274998 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 19 FOLLOWER]: Advancing to term 20
I1014 19:25:48.276829 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.277405 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 20 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.277660 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 20 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.278375 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.278404 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.278568 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 20 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.278620 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 20 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.278980 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 20 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 20 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.280150 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 20 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 20 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.280292 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost.
I1014 19:25:48.280287 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 19. Current term is 20. Ops: []
I1014 19:25:48.280618 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:25:48.280728 29813 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I1014 19:25:48.280894 29494 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.281159 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Stepping down as leader of term 19
I1014 19:25:48.281216 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:48.281400 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.281549 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:48.281591 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:48.281687 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 19 FOLLOWER]: Advancing to term 20
I1014 19:25:48.366011 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:48.366094 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:25:48.366153 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 19 FOLLOWER]: Advancing to term 20
I1014 19:25:48.368017 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.368366 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 20 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.368569 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 20 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.368999 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.369189 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (vote granted)
I1014 19:25:48.369905 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.370117 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 20: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:48.370566 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 20 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 20: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:48.371325 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 20.
I1014 19:25:48.371670 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 20 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.371737 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won.
I1014 19:25:48.372068 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:25:48.372145 30244 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 FOLLOWER]: Leader election won for term 20
I1014 19:25:48.372248 30244 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:48.372516 30244 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.374044 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 20 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.374289 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.374742 30525 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:48.376775 30525 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:48.416846 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.416951 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.179s (starting election)
I1014 19:25:48.417050 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 20 FOLLOWER]: Advancing to term 21
I1014 19:25:48.419287 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.419796 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 21 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.419962 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 21 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.420645 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.420652 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.420809 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 21 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.420800 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 20 FOLLOWER]: Advancing to term 21
I1014 19:25:48.420882 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:25:48.423020 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 21.
I1014 19:25:48.423418 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 21 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.423523 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won.
I1014 19:25:48.423792 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 21 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 21 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.424345 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.155s (election complete)
I1014 19:25:48.424469 29813 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 FOLLOWER]: Leader election won for term 21
I1014 19:25:48.424664 29813 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.424844 29813 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.425804 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 21 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.426014 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.454984 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 21 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 20. Current term is 21. Ops: []
I1014 19:25:48.454988 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 20. Current term is 21. Ops: []
I1014 19:25:48.455796 30244 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.456506 30244 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 LEADER]: Stepping down as leader of term 20
I1014 19:25:48.456511 29049 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.456602 30244 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:48.456732 29049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 21.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I1014 19:25:48.456928 30244 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.457062 30244 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:48.457126 30244 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.457255 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 20 FOLLOWER]: Advancing to term 21
I1014 19:25:48.528251 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 21 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I1014 19:25:48.528852 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.529284 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.529379 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:25:48.529515 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 21 FOLLOWER]: Advancing to term 22
I1014 19:25:48.531266 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.531698 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 22 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.531875 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 22 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.532173 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 22 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 21. Current term is 22. Ops: []
I1014 19:25:48.532330 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.532518 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 21 FOLLOWER]: Advancing to term 22
I1014 19:25:48.532539 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.532773 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 22 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.532929 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.533174 29813 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Stepping down as leader of term 21
I1014 19:25:48.533215 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 22 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 22 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.533258 29813 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.533560 29813 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.533727 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:48.533803 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.533919 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 21 FOLLOWER]: Advancing to term 22
I1014 19:25:48.534520 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I1014 19:25:48.534906 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 22 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I1014 19:25:48.534991 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost.
I1014 19:25:48.535223 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:25:48.535290 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I1014 19:25:48.551728 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.551811 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (starting election)
I1014 19:25:48.551897 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 22 FOLLOWER]: Advancing to term 23
I1014 19:25:48.553823 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.554205 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 23 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.554406 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 23 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.554869 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.555053 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 22 FOLLOWER]: Advancing to term 23
I1014 19:25:48.555035 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.555130 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.211s (vote granted)
I1014 19:25:48.555184 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 22 FOLLOWER]: Advancing to term 23
I1014 19:25:48.557248 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 23.
I1014 19:25:48.557570 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I1014 19:25:48.557790 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 23 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.557864 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won.
I1014 19:25:48.558081 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 23 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I1014 19:25:48.558198 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:25:48.558300 30244 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 FOLLOWER]: Leader election won for term 23
I1014 19:25:48.558440 30244 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:48.558647 30244 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.559494 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 23 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.559702 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.620759 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.620847 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:25:48.620932 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 23 FOLLOWER]: Advancing to term 24
I1014 19:25:48.623183 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.623721 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 24 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.623921 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 24 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.624630 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.624680 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.624838 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 24 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.624855 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 23 FOLLOWER]: Advancing to term 24
I1014 19:25:48.624935 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (vote granted)
I1014 19:25:48.625223 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 24 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 24 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.627089 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 24.
I1014 19:25:48.627449 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 24 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.627517 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won.
I1014 19:25:48.627766 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:25:48.627841 29813 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 FOLLOWER]: Leader election won for term 24
I1014 19:25:48.627949 29813 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.628170 29813 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.629179 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 24 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.629488 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.638519 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 23. Current term is 24. Ops: []
I1014 19:25:48.638952 29049 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.639160 29049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I1014 19:25:48.639271 29049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 LEADER]: Stepping down as leader of term 23
I1014 19:25:48.639338 29049 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:48.639550 29049 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.639748 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:48.639804 29049 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.639932 29049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 23 FOLLOWER]: Advancing to term 24
I1014 19:25:48.730618 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I1014 19:25:48.730698 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 16
I1014 19:25:48.730736 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 23.17
I1014 19:25:48.730804 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:48.731467 29813 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.731649 29813 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 20.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I1014 19:25:48.732370 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 15
I1014 19:25:48.732424 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 20.16
I1014 19:25:48.732488 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:48.733065 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 24 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.733403 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.734201 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 24 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I1014 19:25:48.734947 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.735393 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.735515 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:25:48.735399 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 24 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.735620 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 24 FOLLOWER]: Advancing to term 25
I1014 19:25:48.735662 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.737859 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.738204 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 25 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 24. Current term is 25. Ops: []
I1014 19:25:48.738440 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 25 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.738766 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 25 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.739012 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 25 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.739464 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.739563 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 25 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.739835 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 25 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.740294 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 25 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 25 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.740793 29813 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Stepping down as leader of term 24
I1014 19:25:48.740856 29813 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.741055 29813 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.741250 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:48.741317 29813 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.741425 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 24 FOLLOWER]: Advancing to term 25
I1014 19:25:48.744741 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 25 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:48.744737 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 24 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.745079 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.744737 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 24 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.745201 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.745275 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 25 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 25 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:48.745385 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost.
I1014 19:25:48.745662 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.211s (election complete)
I1014 19:25:48.745728 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I1014 19:25:48.832000 29813 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:48.832108 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (starting election)
I1014 19:25:48.832193 29813 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 25 FOLLOWER]: Advancing to term 26
I1014 19:25:48.833853 29813 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.834385 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 26 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.834609 29813 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 26 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.835170 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.835189 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.835338 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 25 FOLLOWER]: Advancing to term 26
I1014 19:25:48.835391 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 26 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.835423 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.166s (vote granted)
I1014 19:25:48.835850 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 26 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 26 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.836215 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:48.836289 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:25:48.836359 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 24 FOLLOWER]: Advancing to term 25
I1014 19:25:48.837164 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 26.
I1014 19:25:48.837488 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 26 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.837563 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won.
I1014 19:25:48.837826 29813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:25:48.837898 29813 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 FOLLOWER]: Leader election won for term 26
I1014 19:25:48.838043 29813 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.838222 29813 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.838415 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.838860 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 25 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.839004 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 25 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.839056 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.839373 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.839781 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.839951 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 25 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.839906 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.840080 28640 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 25. Current term is 26.
W1014 19:25:48.840502 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 25 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 25 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.840554 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W1014 19:25:48.840703 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 25 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 25. Current term is 26.
I1014 19:25:48.840785 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:25:48.840898 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 25 FOLLOWER]: Advancing to term 26
I1014 19:25:48.842792 30244 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 25 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.870263 29579 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:48.871172 29579 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:48.873607 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I1014 19:25:48.873617 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 26 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I1014 19:25:48.874279 29813 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I1014 19:25:48.874526 30652 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I1014 19:25:48.876144 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.876543 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.878221 30653 mvcc.cc:203] Tried to move safe_time back from 6306080148980326400 to 6306080148843532288. Current Snapshot: MvccSnapshot[committed={T|T < 6306080148980326400}]
I1014 19:25:48.878116 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.878342 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.878516 29579 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:48.880209 29579 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:48.883317 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.883620 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.884857 30685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.884973 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.885748 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.885649 29813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.885870 29813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:48.885957 30685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.885294 30684 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.886359 30684 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.886900 30680 mvcc.cc:203] Tried to move safe_time back from 6306080148980326400 to 6306080148843532288. Current Snapshot: MvccSnapshot[committed={T|T < 6306080148980326400}]
I1014 19:25:48.887485 30684 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 26 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:48.887713 30684 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:48.978509 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:48.978615 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:25:48.978688 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 26 FOLLOWER]: Advancing to term 27
I1014 19:25:48.981099 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.982215 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 27 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:48.982672 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 27 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.982916 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:48.983114 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 27 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.983304 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.983501 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 27 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.983712 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 27 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 27 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.983901 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 27 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 27 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:48.983976 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost.
I1014 19:25:48.984189 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:25:48.984293 30244 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I1014 19:25:48.987686 30684 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:48.987776 30684 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:25:48.987848 30684 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 26 FOLLOWER]: Advancing to term 27
I1014 19:25:48.990022 30684 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.990530 30684 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 27 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:48.990703 30684 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 27 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:48.991262 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 26. Current term is 27. Ops: []
I1014 19:25:48.991396 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 26. Current term is 27. Ops: []
I1014 19:25:48.991302 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:48.991628 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 27 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:48.992169 30652 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I1014 19:25:48.992506 30652 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Stepping down as leader of term 26
I1014 19:25:48.992610 30652 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:48.992568 29813 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I1014 19:25:48.992909 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 27 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 27 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:48.993451 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:48.993881 30652 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:48.994117 30652 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:48.994199 30652 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:48.994343 30652 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 26 FOLLOWER]: Advancing to term 27
I1014 19:25:48.998950 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (vote granted)
I1014 19:25:49.000967 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 27.
I1014 19:25:49.001446 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 27 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.001549 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won.
I1014 19:25:49.001917 30684 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:25:49.001991 30684 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 FOLLOWER]: Leader election won for term 27
I1014 19:25:49.002130 30684 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:49.002380 30684 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.003602 30685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.003800 30685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.088696 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 27 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I1014 19:25:49.089211 30685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I1014 19:25:49.090597 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.091528 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.091500 30684 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.091708 30684 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.092033 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I1014 19:25:49.092500 30685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I1014 19:25:49.092573 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.092785 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.094063 29049 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.094312 29049 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.094393 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.094594 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.192612 30652 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:49.192721 30652 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:49.192817 30652 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 27 FOLLOWER]: Advancing to term 28
I1014 19:25:49.194434 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:49.194581 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:25:49.194692 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 27 FOLLOWER]: Advancing to term 28
I1014 19:25:49.194702 30652 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.195255 30652 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 28 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.195463 30652 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 28 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.196460 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.196524 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:49.196689 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 28 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.197106 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 28 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 28 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.197146 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.197515 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 28: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:49.197679 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 28 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.197873 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 28 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.198055 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 28 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 28: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:49.198184 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost.
I1014 19:25:49.198447 30652 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:25:49.198544 30652 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I1014 19:25:49.198690 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.198743 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:49.198849 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 28 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.198875 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 28: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:49.199221 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 28 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 28: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:49.199362 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 28 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 28 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.199436 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost.
I1014 19:25:49.199676 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (election complete)
I1014 19:25:49.199800 30244 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I1014 19:25:49.202358 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 27 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.202580 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.203218 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 27. Current term is 28. Ops: []
I1014 19:25:49.203766 29177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I1014 19:25:49.203995 30685 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Stepping down as leader of term 27
I1014 19:25:49.204051 30685 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:49.204293 30685 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.204547 30685 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:49.204607 30685 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:49.204748 30685 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 27 FOLLOWER]: Advancing to term 28
I1014 19:25:49.290621 30685 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:49.290709 30685 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:25:49.290782 30685 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 28 FOLLOWER]: Advancing to term 29
I1014 19:25:49.292300 30685 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.292771 30685 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 29 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.292959 30685 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 29 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.293478 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:49.293669 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 28 FOLLOWER]: Advancing to term 29
I1014 19:25:49.293751 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.211s (vote granted)
I1014 19:25:49.295177 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 29.
I1014 19:25:49.295490 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:49.295563 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 29 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.295646 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won.
I1014 19:25:49.295666 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 28 FOLLOWER]: Advancing to term 29
I1014 19:25:49.295766 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.183s (vote granted)
I1014 19:25:49.295855 30685 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:25:49.295928 30685 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 FOLLOWER]: Leader election won for term 29
I1014 19:25:49.296070 30685 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:49.296260 30685 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.297333 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.297547 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.297616 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 29.
I1014 19:25:49.300681 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 29 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.392571 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 29 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I1014 19:25:49.393129 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I1014 19:25:49.394434 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.394738 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.395594 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.395752 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.396203 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I1014 19:25:49.396179 30244 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.396395 30244 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.396745 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I1014 19:25:49.398193 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.398536 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.398378 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.398681 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.478372 30685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.478602 30685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.496613 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 29 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:49.496719 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (starting election)
I1014 19:25:49.496796 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 29 FOLLOWER]: Advancing to term 30
I1014 19:25:49.498281 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.498697 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 30 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.498898 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 30 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.499379 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.499613 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 30 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.499662 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:49.499874 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 30 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.500049 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 30 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 30 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.500224 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 30 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 30 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.500305 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost.
I1014 19:25:49.500526 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (election complete)
I1014 19:25:49.500604 30244 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I1014 19:25:49.511751 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 29. Current term is 30. Ops: []
I1014 19:25:49.512265 30685 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I1014 19:25:49.512542 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 LEADER]: Stepping down as leader of term 29
I1014 19:25:49.512610 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:49.512795 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.513010 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:49.513048 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:49.513152 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 29 FOLLOWER]: Advancing to term 30
I1014 19:25:49.544144 29579 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:49.545147 29579 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:49.546267 29579 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 26 vs 29: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.546586 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.546833 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.547076 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.547353 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.547626 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 29 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.582499 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:49.582592 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:25:49.582650 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 29 FOLLOWER]: Advancing to term 30
I1014 19:25:49.584244 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.584691 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 30 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.584868 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 30 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.585515 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.585697 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:25:49.585671 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:49.585835 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 30: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:49.586238 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 30 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 30: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:49.587779 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 30.
I1014 19:25:49.588838 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 30 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.588924 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won.
I1014 19:25:49.589181 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:25:49.589288 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 FOLLOWER]: Leader election won for term 30
I1014 19:25:49.589448 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:49.589617 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.590734 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 30 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.590873 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.591099 29579 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:49.592741 29579 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:49.605491 30244 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:49.605572 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:49.605640 30244 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 30 FOLLOWER]: Advancing to term 31
I1014 19:25:49.607146 30244 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.607574 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 31 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.607786 30244 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 31 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.608309 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.608465 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 30 FOLLOWER]: Advancing to term 31
I1014 19:25:49.608577 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:25:49.608599 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:49.608817 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 31 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.609167 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 31 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 31 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.610589 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 31.
I1014 19:25:49.610926 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 31 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:49.610981 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won.
I1014 19:25:49.611205 30244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.167s (election complete)
I1014 19:25:49.611294 30244 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 FOLLOWER]: Leader election won for term 31
I1014 19:25:49.611603 30244 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:49.611866 30244 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.612792 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.613011 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.697396 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 31 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 30. Current term is 31. Ops: []
I1014 19:25:49.698000 30652 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I1014 19:25:49.698314 29621 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 LEADER]: Stepping down as leader of term 30
I1014 19:25:49.698364 29621 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:49.698623 29621 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.698789 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:49.698860 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:49.698972 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 30 FOLLOWER]: Advancing to term 31
I1014 19:25:49.710187 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 31 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I1014 19:25:49.710685 31010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I1014 19:25:49.711880 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.712177 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.712911 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.713119 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.713537 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 22. (term mismatch)
I1014 19:25:49.713613 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 21
I1014 19:25:49.713649 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 30.22
I1014 19:25:49.713716 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:49.713829 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.713990 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.714202 31010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I1014 19:25:49.715553 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.715867 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.715553 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.716049 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:49.801028 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 31 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.801337 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:49.906821 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 31 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:49.906919 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (starting election)
I1014 19:25:49.906991 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 31 FOLLOWER]: Advancing to term 32
I1014 19:25:49.908538 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.908890 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 32 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.909087 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 32 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:49.909715 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:49.909713 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:49.909945 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 32 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.909945 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 32 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.910271 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 32 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 32 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.910454 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 32 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 32 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:49.910526 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost.
I1014 19:25:49.910729 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:25:49.910840 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I1014 19:25:49.924226 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 31. Current term is 32. Ops: []
I1014 19:25:49.924768 31033 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I1014 19:25:49.925094 31010 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 LEADER]: Stepping down as leader of term 31
I1014 19:25:49.925155 31010 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:49.925361 31010 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.925554 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:49.925601 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:49.925734 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 31 FOLLOWER]: Advancing to term 32
W1014 19:25:49.988273 29055 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.912s	user 0.001s	sys 0.001s
I1014 19:25:49.988344 29055 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:49.988564 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:49.988654 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:25:49.988729 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 31 FOLLOWER]: Advancing to term 32
I1014 19:25:49.989374 29055 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:49.990703 29055 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 19 vs 32: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.990993 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.991062 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
W1014 19:25:49.991232 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.991502 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.991634 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 32 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
W1014 19:25:49.991739 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:49.991958 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.992192 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 32 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:25:49.992470 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.992871 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:49.992920 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:49.993093 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (vote granted)
I1014 19:25:49.993211 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 32: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
W1014 19:25:49.993291 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.993654 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 32 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 32: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
W1014 19:25:49.993999 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:49.995226 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 32.
I1014 19:25:49.995723 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 32 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:49.995785 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won.
I1014 19:25:49.996031 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:25:49.996104 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 FOLLOWER]: Leader election won for term 32
I1014 19:25:49.996253 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:49.998378 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:49.999915 30652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 32 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.000152 30652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.036705 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.036801 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.174s (starting election)
I1014 19:25:50.036878 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 32 FOLLOWER]: Advancing to term 33
I1014 19:25:50.038372 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.038823 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 33 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.039062 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 33 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.039649 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.039788 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 32 FOLLOWER]: Advancing to term 33
I1014 19:25:50.039878 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (vote granted)
I1014 19:25:50.039870 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.040040 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 33 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.040390 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 33 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 33 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.041463 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 33.
I1014 19:25:50.041779 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 33 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.041841 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won.
I1014 19:25:50.042050 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.214s (election complete)
I1014 19:25:50.042138 29177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 FOLLOWER]: Leader election won for term 33
I1014 19:25:50.042501 29177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.042851 29177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.043818 31154 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.044070 31154 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.044281 29055 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:50.045897 29055 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:50.079802 29579 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:50.080871 29579 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:50.083221 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 32. Current term is 33. Ops: []
I1014 19:25:50.083307 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 33 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 32. Current term is 33. Ops: []
I1014 19:25:50.083887 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I1014 19:25:50.084398 29621 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 33.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I1014 19:25:50.084686 30652 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I1014 19:25:50.084928 31155 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 LEADER]: Stepping down as leader of term 32
I1014 19:25:50.085022 31155 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:50.085288 31155 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.24, Last appended by leader: 24, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.085448 31155 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:50.085559 31155 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:50.085681 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 32 FOLLOWER]: Advancing to term 33
I1014 19:25:50.120770 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 33 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I1014 19:25:50.121295 31154 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I1014 19:25:50.122689 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.122918 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.124415 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 33 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 33 index: 23. (term mismatch)
I1014 19:25:50.124514 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 22
I1014 19:25:50.124276 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.124552 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 32.23
I1014 19:25:50.124579 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.124617 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:50.124686 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted WRITE_OP operation due to leader change: 32.24
I1014 19:25:50.124680 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.124897 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.125159 29579 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 30 vs 33: Aborted: Transaction aborted by new leader
W1014 19:25:50.125442 29579 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 33 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.125495 31159 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I1014 19:25:50.126804 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.127053 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.127259 31155 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.127441 31155 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.225349 31010 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:50.225473 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (starting election)
I1014 19:25:50.225566 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 33 FOLLOWER]: Advancing to term 34
I1014 19:25:50.227181 31010 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.227458 31155 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:50.227564 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (starting election)
I1014 19:25:50.227640 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 33 FOLLOWER]: Advancing to term 34
I1014 19:25:50.227744 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 34 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.227963 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 34 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.228726 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.228917 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 34 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.228922 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.229342 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 34 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 34 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.230720 29055 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:50.230661 31155 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.231098 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 34 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.231272 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 34 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.231564 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 34: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:50.231626 29055 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:50.232208 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 34 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 34: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:50.232316 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost.
I1014 19:25:50.232226 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.232518 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 34: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:50.232722 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:25:50.232784 31010 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I1014 19:25:50.233065 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 34 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 34: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:50.233661 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.233893 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 34 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.234208 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 34 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 34 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.234292 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost.
I1014 19:25:50.234442 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 33. Current term is 34. Ops: []
I1014 19:25:50.234346 31154 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 33 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.234568 31154 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.234601 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:25:50.234659 31155 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I1014 19:25:50.234917 31159 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I1014 19:25:50.235121 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Stepping down as leader of term 33
I1014 19:25:50.235184 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.235226 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 33. Current term is 34. Ops: []
I1014 19:25:50.235397 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.235599 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:50.235692 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:50.235826 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 33 FOLLOWER]: Advancing to term 34
W1014 19:25:50.337754 30525 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.961s	user 0.001s	sys 0.000s
I1014 19:25:50.337817 30525 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:50.338799 30525 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:50.340548 30525 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 20 vs 34: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 34 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:50.341409 30525 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 34 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:50.341677 30525 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 34 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:50.341934 30525 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 34 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:50.342145 30525 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 34 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.351250 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.351331 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (starting election)
I1014 19:25:50.351387 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 34 FOLLOWER]: Advancing to term 35
I1014 19:25:50.353102 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.353704 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 35 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.353981 31155 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.353999 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 35 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.354058 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:50.354140 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 34 FOLLOWER]: Advancing to term 35
I1014 19:25:50.354660 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.355082 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.355247 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 34 FOLLOWER]: Advancing to term 35
I1014 19:25:50.355326 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.175s (vote granted)
I1014 19:25:50.356058 31155 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.356374 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 35: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:50.356544 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 35 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.356734 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 35 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.357396 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.357414 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 35 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 35: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:50.357625 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 35.
I1014 19:25:50.357566 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.357743 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 35: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:50.357872 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 35 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:50.358023 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 35 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.358122 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won.
I1014 19:25:50.358199 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 35 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 35 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:50.358358 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 35 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 35: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:50.358427 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost.
I1014 19:25:50.358438 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:25:50.358513 29177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 FOLLOWER]: Leader election won for term 35
I1014 19:25:50.358645 29177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.358666 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.223s (election complete)
I1014 19:25:50.358728 31155 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I1014 19:25:50.358845 29177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.360095 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.360286 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.445253 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I1014 19:25:50.445807 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I1014 19:25:50.447717 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.447919 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.449199 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.449404 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.449659 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I1014 19:25:50.450605 29055 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:50.452450 29055 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:50.452725 31159 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I1014 19:25:50.454641 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.454864 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.455271 31155 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.455271 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 35 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.455490 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.455655 31155 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.549471 31010 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:50.549568 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:25:50.549652 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 35 FOLLOWER]: Advancing to term 36
I1014 19:25:50.551218 31010 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.551652 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 36 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.551836 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 36 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.552358 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.552575 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.552462 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.552680 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.552991 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 36 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.553146 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 36 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.553223 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost.
I1014 19:25:50.553462 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:25:50.553530 31010 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I1014 19:25:50.553890 31155 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:50.553972 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:25:50.554041 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 35 FOLLOWER]: Advancing to term 36
I1014 19:25:50.555548 31155 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.555914 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 36 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.556104 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 36 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.556459 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.556649 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.556602 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.556759 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 36: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:50.557013 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 36 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 36 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.557170 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 36 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 36: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:50.557271 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost.
I1014 19:25:50.557621 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:25:50.557694 31155 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I1014 19:25:50.566861 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 35. Current term is 36. Ops: []
I1014 19:25:50.567286 31159 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I1014 19:25:50.567565 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Stepping down as leader of term 35
I1014 19:25:50.567616 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.567816 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.567965 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:50.568022 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:50.568125 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 35 FOLLOWER]: Advancing to term 36
I1014 19:25:50.643640 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.643720 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:25:50.643796 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 36 FOLLOWER]: Advancing to term 37
I1014 19:25:50.645633 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.646132 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 37 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.646364 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 37 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.647363 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.647586 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 36 FOLLOWER]: Advancing to term 37
I1014 19:25:50.647528 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.647693 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
I1014 19:25:50.647697 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 36 FOLLOWER]: Advancing to term 37
I1014 19:25:50.647800 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (vote granted)
I1014 19:25:50.649893 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 37.
I1014 19:25:50.650228 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 37.
I1014 19:25:50.650290 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 37 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.650365 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won.
I1014 19:25:50.650637 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:25:50.650735 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 37 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.650815 29177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 FOLLOWER]: Leader election won for term 37
I1014 19:25:50.651023 29177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.651252 29177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.652297 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 37 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.652549 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.761564 31155 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.761693 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.223s (starting election)
I1014 19:25:50.761783 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 37 FOLLOWER]: Advancing to term 38
I1014 19:25:50.764202 31155 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.764777 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 38 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:50.765501 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 37. Current term is 38. Ops: []
I1014 19:25:50.765458 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:50.765640 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 38 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.765991 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 38 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 38 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.766088 29177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I1014 19:25:50.766227 31155 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 38 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.766335 31159 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 LEADER]: Stepping down as leader of term 37
I1014 19:25:50.766381 31159 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.766676 31159 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.766820 31159 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:50.766865 31159 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:50.766944 31159 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 37 FOLLOWER]: Advancing to term 38
I1014 19:25:50.766914 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.767073 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 37 FOLLOWER]: Advancing to term 38
I1014 19:25:50.767161 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.213s (vote granted)
I1014 19:25:50.769479 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 38.
I1014 19:25:50.769774 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 38 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.769839 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won.
I1014 19:25:50.770076 31155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.222s (election complete)
I1014 19:25:50.770148 31155 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 FOLLOWER]: Leader election won for term 38
I1014 19:25:50.770295 31155 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:50.770514 31155 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.771358 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 38 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.771566 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.772100 31406 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:50.773666 31406 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:50.872133 31159 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:50.872216 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:25:50.872294 31159 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 38 FOLLOWER]: Advancing to term 39
I1014 19:25:50.873960 31159 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.874493 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 39 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.874703 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 39 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:50.875149 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:50.875311 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 38 FOLLOWER]: Advancing to term 39
I1014 19:25:50.875406 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (vote granted)
I1014 19:25:50.877410 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:50.877496 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 39.
I1014 19:25:50.877605 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 39 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.877920 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 39 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:50.877985 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won.
I1014 19:25:50.878211 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:25:50.878309 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 39 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 39 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:50.878309 31159 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 FOLLOWER]: Leader election won for term 39
I1014 19:25:50.878583 31159 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:50.878759 31159 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.879735 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.880064 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.880348 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 38. Current term is 39. Ops: []
I1014 19:25:50.880919 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I1014 19:25:50.881132 29621 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 39.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I1014 19:25:50.881261 31155 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 LEADER]: Stepping down as leader of term 38
I1014 19:25:50.881314 31155 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:50.881515 31155 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:50.881716 31155 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:50.881808 31155 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:50.881917 31155 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 38 FOLLOWER]: Advancing to term 39
I1014 19:25:50.962079 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 39 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I1014 19:25:50.962631 31159 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I1014 19:25:50.962785 31159 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 38.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I1014 19:25:50.964010 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 25
I1014 19:25:50.964054 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 38.26
I1014 19:25:50.964128 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:50.964953 31155 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.965225 31155 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.966936 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.967159 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:50.967713 31155 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.967819 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 39 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I1014 19:25:50.967849 31155 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.967968 31155 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.968128 31155 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.968441 31159 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I1014 19:25:50.985922 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.986217 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.986093 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.986364 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 39 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:50.987251 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:50.987427 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.067507 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:51.067608 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:51.067704 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 39 FOLLOWER]: Advancing to term 40
I1014 19:25:51.069274 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.069720 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 40 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.069927 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 40 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.070322 31534 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 39 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:51.070451 31534 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:25:51.070654 31534 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 39 FOLLOWER]: Advancing to term 40
I1014 19:25:51.070920 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.071132 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 40 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.071630 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 40 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 40 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.072691 31534 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.070935 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.073081 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 40: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:51.073170 31534 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 40 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.073385 31534 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 40 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.073735 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.073880 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 40 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.074044 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.074232 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 40: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:51.074242 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 40 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 40 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.074589 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 40 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 40: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:51.074645 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 40 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 40: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:51.074676 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost.
I1014 19:25:51.074733 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost.
I1014 19:25:51.074918 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:25:51.074987 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I1014 19:25:51.075047 31534 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:25:51.075111 31534 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I1014 19:25:51.082810 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 39. Current term is 40. Ops: []
I1014 19:25:51.083294 31159 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I1014 19:25:51.083568 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Stepping down as leader of term 39
I1014 19:25:51.083633 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:51.083837 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.083998 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:51.084054 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:51.084192 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 39 FOLLOWER]: Advancing to term 40
I1014 19:25:51.110448 29055 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:51.111464 29055 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:51.112592 29055 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 35 vs 40: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.112884 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.113238 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.113478 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.113773 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.114006 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:51.114245 29055 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 40 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.191634 31534 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:51.191720 31534 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (starting election)
I1014 19:25:51.191807 31534 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 40 FOLLOWER]: Advancing to term 41
I1014 19:25:51.193506 31534 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.193928 31534 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 41 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.194126 31534 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 41 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.194607 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.194805 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 40 FOLLOWER]: Advancing to term 41
I1014 19:25:51.194772 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.194883 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (vote granted)
I1014 19:25:51.194952 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 40 FOLLOWER]: Advancing to term 41
I1014 19:25:51.195045 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.168s (vote granted)
I1014 19:25:51.197041 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 41.
I1014 19:25:51.197167 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 41.
I1014 19:25:51.197532 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 41 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.197638 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won.
I1014 19:25:51.197780 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 41 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.197891 31534 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:25:51.197957 31534 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 FOLLOWER]: Leader election won for term 41
I1014 19:25:51.198089 31534 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:51.198351 31534 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.199633 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.199846 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.200417 31545 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:51.202533 31545 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:51.284795 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 41 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I1014 19:25:51.285276 31533 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I1014 19:25:51.286535 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.286792 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.288883 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 41 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I1014 19:25:51.289294 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.289497 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.289921 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.290096 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.290254 31010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I1014 19:25:51.291796 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.292089 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.291792 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.292317 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.389010 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 41 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:51.389101 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (starting election)
I1014 19:25:51.389180 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 41 FOLLOWER]: Advancing to term 42
I1014 19:25:51.390780 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.391206 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 42 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.391369 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 42 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.391746 31159 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 41 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:51.391826 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:51.391930 31159 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 41 FOLLOWER]: Advancing to term 42
I1014 19:25:51.392067 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.392298 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 42 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.392776 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 42 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 42 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.393352 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.394011 31159 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.394310 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 42: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:51.394548 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 42 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.394697 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 42 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.394788 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 42 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 42: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:51.394865 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost.
I1014 19:25:51.395103 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:25:51.395171 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I1014 19:25:51.395436 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.395607 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 42 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.395781 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.395951 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 42 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 42 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.395977 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 42: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:51.396306 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 42 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 42: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:51.396371 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost.
I1014 19:25:51.396608 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (election complete)
I1014 19:25:51.396669 31159 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I1014 19:25:51.400637 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 41 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.400867 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.401060 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 41. Current term is 42. Ops: []
I1014 19:25:51.401531 31010 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I1014 19:25:51.401768 31010 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Stepping down as leader of term 41
I1014 19:25:51.401819 31010 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:51.402016 31010 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.402163 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:51.402246 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:51.402348 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 41 FOLLOWER]: Advancing to term 42
I1014 19:25:51.521761 31010 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:51.521844 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:25:51.521914 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 42 FOLLOWER]: Advancing to term 43
I1014 19:25:51.523995 31010 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.524516 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 43 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.524677 31010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 43 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.525193 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.525241 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.525354 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 42 FOLLOWER]: Advancing to term 43
I1014 19:25:51.525391 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 42 FOLLOWER]: Advancing to term 43
I1014 19:25:51.525429 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.214s (vote granted)
I1014 19:25:51.525461 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.176s (vote granted)
I1014 19:25:51.527035 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 43.
I1014 19:25:51.527345 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 43 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.527412 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 43 election: Election decided. Result: candidate won.
I1014 19:25:51.527436 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 43.
I1014 19:25:51.527678 31010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:25:51.527828 31010 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 FOLLOWER]: Leader election won for term 43
I1014 19:25:51.527825 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 43 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.528064 31010 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:51.528229 31010 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.529111 31534 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.529386 31534 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.626143 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I1014 19:25:51.626777 31010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I1014 19:25:51.628114 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.628340 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.629180 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 43 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I1014 19:25:51.629585 31534 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.629772 31534 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.630013 31533 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I1014 19:25:51.631350 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.631620 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.631675 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.631937 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.632023 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.632261 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.731573 31159 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 43 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:51.731662 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (starting election)
I1014 19:25:51.731734 31159 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 43 FOLLOWER]: Advancing to term 44
I1014 19:25:51.733333 31159 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.733826 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 44 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.734041 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 44 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.734597 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.734757 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 44 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.734717 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.734947 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 44 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.735092 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 44 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 44 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.735296 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 44 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 44 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.735391 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost.
I1014 19:25:51.735654 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:25:51.735749 31159 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I1014 19:25:51.742260 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 43 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.742566 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.742718 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 43. Current term is 44. Ops: []
I1014 19:25:51.743209 31533 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 43.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I1014 19:25:51.743456 31010 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 LEADER]: Stepping down as leader of term 43
I1014 19:25:51.743523 31010 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:51.743703 31010 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.743927 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:51.743965 31010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:51.744065 31010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 43 FOLLOWER]: Advancing to term 44
I1014 19:25:51.830852 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:51.830967 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:25:51.831053 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 43 FOLLOWER]: Advancing to term 44
I1014 19:25:51.833556 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.834121 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 44 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:51.834359 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 44 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.834991 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:51.835011 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.835172 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (vote granted)
I1014 19:25:51.835204 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 44: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:51.835608 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 44 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 44: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:51.837195 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 44.
I1014 19:25:51.837502 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 44 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.837564 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won.
I1014 19:25:51.837785 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:25:51.837864 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 FOLLOWER]: Leader election won for term 44
I1014 19:25:51.838230 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:51.838526 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.839686 31808 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 44 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.840072 31808 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.851025 31159 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:51.851141 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:25:51.851233 31159 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 44 FOLLOWER]: Advancing to term 45
I1014 19:25:51.852854 31159 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.853374 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 45 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.853610 31159 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 45 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:51.854128 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:51.854297 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 44 FOLLOWER]: Advancing to term 45
I1014 19:25:51.854377 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (vote granted)
I1014 19:25:51.856145 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 45.
I1014 19:25:51.856546 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 45 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:51.856468 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:51.856623 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won.
I1014 19:25:51.856737 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 45 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.856858 31159 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.154s (election complete)
I1014 19:25:51.856928 31159 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 FOLLOWER]: Leader election won for term 45
I1014 19:25:51.857079 31159 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:51.857277 31159 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.858252 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.858471 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 45 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 45 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:51.858582 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.859119 31832 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:51.861222 31832 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:51.915626 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 44. Current term is 45. Ops: []
I1014 19:25:51.920001 31808 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I1014 19:25:51.920289 29621 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 LEADER]: Stepping down as leader of term 44
I1014 19:25:51.920358 29621 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:51.920636 29621 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.30, Last appended by leader: 30, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:51.920786 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:51.920855 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:51.920974 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 44 FOLLOWER]: Advancing to term 45
I1014 19:25:51.962225 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 45 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 45 index: 30. (term mismatch)
I1014 19:25:51.962303 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 29
I1014 19:25:51.962348 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 44.30
I1014 19:25:51.962411 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:51.963199 29177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I1014 19:25:51.964664 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.964926 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.965965 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.966161 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:51.966488 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I1014 19:25:51.966614 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.966892 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.967128 31159 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I1014 19:25:51.968474 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.968852 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:51.968763 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:51.968955 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.046532 31159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 45 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.046893 31159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.067224 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:52.067358 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:52.067473 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 45 FOLLOWER]: Advancing to term 46
I1014 19:25:52.069743 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.070283 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 46 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.070559 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 46 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:52.071074 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.071307 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 46 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.071740 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 46 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 46 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.072319 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:52.072504 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 46 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.072914 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 46 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 46 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.072985 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost.
I1014 19:25:52.073230 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:25:52.073308 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I1014 19:25:52.076562 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 45. Current term is 46. Ops: []
I1014 19:25:52.077095 31911 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 45.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I1014 19:25:52.077409 29177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 LEADER]: Stepping down as leader of term 45
I1014 19:25:52.077476 29177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:52.077735 29177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.077952 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:52.078025 29177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:52.078194 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 45 FOLLOWER]: Advancing to term 46
I1014 19:25:52.147563 31533 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:52.147678 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:25:52.147758 31533 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 45 FOLLOWER]: Advancing to term 46
I1014 19:25:52.149394 31533 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.149847 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 46 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.150038 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 46 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.150722 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.150871 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:52.150918 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (vote granted)
I1014 19:25:52.151080 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 46: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:52.151491 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 46 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 46: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:52.153095 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 46.
I1014 19:25:52.153461 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 46 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.153534 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won.
I1014 19:25:52.153820 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:25:52.153905 31533 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 FOLLOWER]: Leader election won for term 46
I1014 19:25:52.154043 31533 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:52.154279 31533 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.155483 31010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 46 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.155706 31010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.198354 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:52.198448 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (starting election)
I1014 19:25:52.198540 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 46 FOLLOWER]: Advancing to term 47
I1014 19:25:52.200084 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.200536 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 47 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.200716 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 47 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:52.201318 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.201349 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:52.201516 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 46 FOLLOWER]: Advancing to term 47
I1014 19:25:52.201544 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 47 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.201609 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:25:52.201885 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 47 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 47 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.203716 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 47.
I1014 19:25:52.204088 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 47 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.204174 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won.
I1014 19:25:52.204413 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.162s (election complete)
I1014 19:25:52.204493 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 FOLLOWER]: Leader election won for term 47
I1014 19:25:52.204658 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:52.204857 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.206075 31808 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 47 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.206240 31808 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.236505 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 47 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 46. Current term is 47. Ops: []
I1014 19:25:52.237104 31010 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I1014 19:25:52.237360 31533 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 LEADER]: Stepping down as leader of term 46
I1014 19:25:52.237432 31533 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:52.237711 31533 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 46.31, Last appended by leader: 31, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.237901 31533 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:52.237954 31533 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:52.238081 31533 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 46 FOLLOWER]: Advancing to term 47
I1014 19:25:52.311061 29177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:52.311177 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:25:52.311262 29177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 47 FOLLOWER]: Advancing to term 48
I1014 19:25:52.311722 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 46 index: 31. Preceding OpId from leader: term: 47 index: 31. (term mismatch)
I1014 19:25:52.311782 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 30
I1014 19:25:52.311827 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 46.31
I1014 19:25:52.311870 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:52.312418 31808 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I1014 19:25:52.313496 29177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.313721 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 47 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.314002 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.313999 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 48 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:52.314157 29177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 48 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.315032 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 48 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:52.314909 31808 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 47 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.315240 31808 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.315294 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 48 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.315807 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 48 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 48 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.316015 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 48 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:52.316267 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 48 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.316269 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 47. Current term is 48. Ops: []
I1014 19:25:52.316691 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 48 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 48 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.316776 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost.
I1014 19:25:52.317018 29177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:25:52.316874 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 47 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.317083 29177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I1014 19:25:52.317147 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.317688 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I1014 19:25:52.317916 31808 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 LEADER]: Stepping down as leader of term 47
I1014 19:25:52.317971 31808 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:52.318192 31808 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.318336 31808 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:52.318424 31808 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:52.318549 31808 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 47 FOLLOWER]: Advancing to term 48
I1014 19:25:52.417750 31533 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:52.417888 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:25:52.417964 31533 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 47 FOLLOWER]: Advancing to term 48
I1014 19:25:52.419725 31533 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.420274 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 48 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.420521 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 48 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.421350 31808 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:52.421350 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:52.421476 31808 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:25:52.421455 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.421584 31808 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 48 FOLLOWER]: Advancing to term 49
I1014 19:25:52.421649 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 48: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:52.422068 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 48 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 48: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:52.423275 31808 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.423892 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 48. Current term is 49.
I1014 19:25:52.423892 31808 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 49 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.424124 31808 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 49 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:25:52.424477 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 48 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 48. Current term is 49.
I1014 19:25:52.424561 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 48 election: Cancelling election due to peer responding with higher term
I1014 19:25:52.424906 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.425043 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 48 FOLLOWER]: Advancing to term 49
I1014 19:25:52.425134 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (vote granted)
I1014 19:25:52.425104 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:52.425331 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 48 FOLLOWER]: Advancing to term 49
I1014 19:25:52.425407 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (vote granted)
I1014 19:25:52.426890 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 49.
I1014 19:25:52.427029 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:25:52.427211 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 49.
I1014 19:25:52.427242 31533 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 49 FOLLOWER]: Leader election lost for term 48. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 48. Current term is 49.
I1014 19:25:52.427292 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 49 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:52.427373 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won.
I1014 19:25:52.427634 31808 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (election complete)
I1014 19:25:52.427716 31808 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Leader election won for term 49
I1014 19:25:52.427865 31808 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:52.428069 31808 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.428642 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 49 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.429493 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.429709 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.524471 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 49 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 49 index: 32. (index mismatch)
I1014 19:25:52.525022 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I1014 19:25:52.527619 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 49 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 49 index: 32. (index mismatch)
I1014 19:25:52.527973 29177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.527973 31911 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.528259 29177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.528332 31911 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.528452 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I1014 19:25:52.529422 31808 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.529664 31808 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.530974 31911 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.531159 31911 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.531191 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.531486 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.533164 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.533151 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 49 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.533417 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.533506 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:25:52.619616 31406 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.846s	user 0.002s	sys 0.000s
I1014 19:25:52.619691 31406 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:52.620877 31406 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:52.627357 31406 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:52.628984 31406 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:52.826691 31533 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 49 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:52.826809 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (starting election)
I1014 19:25:52.826882 31533 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 49 FOLLOWER]: Advancing to term 50
I1014 19:25:52.829061 31533 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.829612 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 50 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:52.830305 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:52.830495 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 50 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.830590 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 49. Current term is 50. Ops: []
I1014 19:25:52.830950 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 50 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 50 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.831329 32135 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 49.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I1014 19:25:52.831449 31533 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 50 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.831691 32135 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 LEADER]: Stepping down as leader of term 49
I1014 19:25:52.831758 32135 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:52.832109 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:52.832059 32135 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 49.33, Last appended by leader: 33, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.832315 32135 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:52.832356 32135 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:52.832487 32135 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 49 FOLLOWER]: Advancing to term 50
I1014 19:25:52.834065 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (vote granted)
I1014 19:25:52.836120 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 50.
I1014 19:25:52.836616 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 50 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.836707 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 50 election: Election decided. Result: candidate won.
I1014 19:25:52.836971 31533 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:25:52.837100 31533 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 FOLLOWER]: Leader election won for term 50
I1014 19:25:52.837364 31533 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:52.837668 31533 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 49.33, Last appended by leader: 33, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.838804 32232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.838966 32232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:25:52.867328 31545 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.665s	user 0.000s	sys 0.001s
I1014 19:25:52.867395 31545 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:52.868381 31545 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:52.870858 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 49 FOLLOWER]: Advancing to term 50
I1014 19:25:52.870867 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 49 index: 33. Preceding OpId from leader: term: 50 index: 35. (index mismatch)
I1014 19:25:52.871498 32232 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I1014 19:25:52.873204 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 50 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 49 index: 33. Preceding OpId from leader: term: 50 index: 35. (index mismatch)
I1014 19:25:52.873275 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.873484 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.873808 32232 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I1014 19:25:52.875018 31545 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:52.875460 31911 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.875701 31911 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.876631 32232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.876864 32232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.878088 31545 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:52.877954 31911 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.878147 31911 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.878834 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.879673 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 50 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:52.879918 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:52.880348 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:52.978350 31911 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 50 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:52.978480 31911 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:52.978556 31911 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 50 FOLLOWER]: Advancing to term 51
I1014 19:25:52.980392 31911 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:52.980907 31911 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 51 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:52.981068 31911 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 51 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:52.981626 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 51 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:52.981654 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 51 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:52.981817 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 51 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.981886 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 51 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.982180 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 51 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 51 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.982364 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 51 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 51 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:52.982445 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost.
I1014 19:25:52.982677 31911 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (election complete)
I1014 19:25:52.982758 31911 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I1014 19:25:52.998726 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 50. Current term is 51. Ops: []
I1014 19:25:52.999256 31533 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 50.35, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I1014 19:25:52.999533 31533 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 LEADER]: Stepping down as leader of term 50
I1014 19:25:52.999583 31533 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:52.999769 31533 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 50.35, Last appended by leader: 35, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.000044 31533 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:53.000093 31533 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:53.000190 31533 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 50 FOLLOWER]: Advancing to term 51
I1014 19:25:53.056125 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:53.056221 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:53.056291 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 50 FOLLOWER]: Advancing to term 51
I1014 19:25:53.057863 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.058274 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 51 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.058459 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 51 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.059084 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 51 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.059279 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 51: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:53.059314 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 51 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.059520 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:25:53.059669 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 51 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 51: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:53.061630 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 51.
I1014 19:25:53.061946 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 51 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.062021 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won.
I1014 19:25:53.062271 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (election complete)
I1014 19:25:53.062356 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 FOLLOWER]: Leader election won for term 51
I1014 19:25:53.062489 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.062690 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 50.35, Last appended by leader: 35, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.063670 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 51 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.063973 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.119379 31911 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:53.119518 31911 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (starting election)
I1014 19:25:53.119592 31911 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 51 FOLLOWER]: Advancing to term 52
I1014 19:25:53.121776 31911 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.122280 31911 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 52 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.123611 31911 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 52 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:53.123577 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 52 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.123773 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 51 FOLLOWER]: Advancing to term 52
I1014 19:25:53.123888 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (vote granted)
I1014 19:25:53.124369 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 52 candidate_status { last_received { term: 50 index: 35 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:53.124626 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 52 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.125133 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 52 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 52 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.125988 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 52.
I1014 19:25:53.126294 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 52 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.126358 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won.
I1014 19:25:53.126628 31911 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.171s (election complete)
I1014 19:25:53.126904 31911 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 FOLLOWER]: Leader election won for term 52
I1014 19:25:53.127591 31911 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:53.127910 31911 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 50.35, Last appended by leader: 35, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.128995 32299 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.129211 32299 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.141705 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 51. Current term is 52. Ops: []
I1014 19:25:53.142309 32135 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I1014 19:25:53.142608 32135 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 LEADER]: Stepping down as leader of term 51
I1014 19:25:53.142663 32135 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.142868 32135 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 51.36, Last appended by leader: 36, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.143010 32135 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:53.143105 32135 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:53.143208 32135 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 51 FOLLOWER]: Advancing to term 52
I1014 19:25:53.217674 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 52 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 51 index: 36. Preceding OpId from leader: term: 52 index: 36. (term mismatch)
I1014 19:25:53.217749 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 35
I1014 19:25:53.217787 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 51.36
I1014 19:25:53.217851 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 52 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:53.218516 32299 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I1014 19:25:53.219790 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.220023 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.221418 32299 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.221618 32299 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.222023 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 50 index: 35. Preceding OpId from leader: term: 52 index: 36. (index mismatch)
I1014 19:25:53.222383 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.222610 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.222672 32413 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I1014 19:25:53.224361 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.224588 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.224716 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.224836 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.315268 32299 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 52 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.315620 32299 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.322993 32135 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:53.323091 32135 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (starting election)
I1014 19:25:53.323177 32135 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 52 FOLLOWER]: Advancing to term 53
I1014 19:25:53.324856 32135 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.325327 32135 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 53 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.325502 32135 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 53 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.326146 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 53 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.326186 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 53 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.326334 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 53 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.326380 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 53 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.326741 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 53 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 53 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.326901 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 53 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 53 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.326967 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost.
I1014 19:25:53.327244 32135 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:25:53.327314 32135 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I1014 19:25:53.334350 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 52. Current term is 53. Ops: []
I1014 19:25:53.334874 32299 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 52.36, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I1014 19:25:53.335145 32413 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 LEADER]: Stepping down as leader of term 52
I1014 19:25:53.335227 32413 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:53.335470 32413 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 52.36, Last appended by leader: 36, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.335665 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:53.335716 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:53.335819 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 52 FOLLOWER]: Advancing to term 53
W1014 19:25:53.387928 31832 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.527s	user 0.001s	sys 0.000s
I1014 19:25:53.388000 31832 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:53.388780 31832 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:53.391089 31832 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 45 vs 53: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:53.391719 31832 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:53.391897 31832 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:53.392102 31832 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:53.392273 31832 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:53.392482 31832 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 53 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.416627 32273 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:53.416736 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:53.416821 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 52 FOLLOWER]: Advancing to term 53
I1014 19:25:53.418740 32273 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.419261 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 53 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.419457 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 53 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:53.420189 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 53 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:53.420189 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 53 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.420408 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 53: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:53.420383 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (vote granted)
I1014 19:25:53.420883 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 53 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 53: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:53.422076 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 53.
I1014 19:25:53.422367 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 53 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.422444 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 53 election: Election decided. Result: candidate won.
I1014 19:25:53.422760 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:25:53.422839 32273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 FOLLOWER]: Leader election won for term 53
I1014 19:25:53.422999 32273 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:53.423187 32273 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 52.36, Last appended by leader: 36, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.424105 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 53 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.424355 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.465299 32135 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:53.465423 32135 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:53.465544 32135 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 53 FOLLOWER]: Advancing to term 54
I1014 19:25:53.467927 32135 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.468566 32135 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 54 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.468765 32135 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 54 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.470841 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 54 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.471024 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 53 FOLLOWER]: Advancing to term 54
I1014 19:25:53.471109 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (vote granted)
I1014 19:25:53.472992 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 54 candidate_status { last_received { term: 52 index: 36 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.473192 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 54 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.473203 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 54.
I1014 19:25:53.473656 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 54 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.473734 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won.
I1014 19:25:53.474004 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 54 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 54 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.474110 32135 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:25:53.474220 32135 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 FOLLOWER]: Leader election won for term 54
I1014 19:25:53.474383 32135 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.474639 32135 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 52.36, Last appended by leader: 36, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.476069 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.476235 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.519788 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 53. Current term is 54. Ops: []
I1014 19:25:53.520364 31533 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I1014 19:25:53.520606 31533 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 54.37 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36
I1014 19:25:53.520747 32273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 LEADER]: Stepping down as leader of term 53
I1014 19:25:53.520802 32273 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:53.520984 32273 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 53.37, Last appended by leader: 37, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.521137 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:53.521244 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:53.521395 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 53 FOLLOWER]: Advancing to term 54
I1014 19:25:53.572051 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 54 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 52 index: 36. Preceding OpId from leader: term: 54 index: 37. (index mismatch)
I1014 19:25:53.572602 29621 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I1014 19:25:53.573985 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.574219 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.575031 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.575278 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.576092 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 54 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 53 index: 37. Preceding OpId from leader: term: 54 index: 37. (term mismatch)
I1014 19:25:53.576153 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 36
I1014 19:25:53.576195 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 53.37
I1014 19:25:53.576246 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:53.576601 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.576823 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.576972 32135 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I1014 19:25:53.578483 31533 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.578721 31533 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.578855 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 54 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.579015 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.676417 32413 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 54 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:53.676555 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:25:53.676628 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 54 FOLLOWER]: Advancing to term 55
I1014 19:25:53.678197 32413 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.678663 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 55 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.678756 32273 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 54 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:53.678838 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:25:53.678891 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 55 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:53.678936 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 54 FOLLOWER]: Advancing to term 55
I1014 19:25:53.679777 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 55 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:53.679978 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 55 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.680197 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 55 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.680418 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 55 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 55 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.681712 32273 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.682066 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 55: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:53.682277 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 55 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.682474 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 55 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:53.682621 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 55 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 55: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:53.682701 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost.
I1014 19:25:53.682986 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (election complete)
I1014 19:25:53.683097 32413 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I1014 19:25:53.683171 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 55 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:53.683341 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 55 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.683450 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 55 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.683629 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 55: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:53.683794 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 55 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 55 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.683980 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 55 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 55: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:53.684067 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost.
I1014 19:25:53.684340 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:25:53.684419 32273 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I1014 19:25:53.687991 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 54. Current term is 55. Ops: []
I1014 19:25:53.688504 32135 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 54.37, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I1014 19:25:53.688787 29621 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Stepping down as leader of term 54
I1014 19:25:53.688848 29621 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.689076 29621 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 54.37, Last appended by leader: 37, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.689235 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:53.689327 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:53.689445 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 54 FOLLOWER]: Advancing to term 55
I1014 19:25:53.766808 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:53.766908 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:53.766991 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 55 FOLLOWER]: Advancing to term 56
I1014 19:25:53.768585 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.768991 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 56 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.769145 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 56 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.769816 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 56 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:53.769860 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 56 candidate_status { last_received { term: 54 index: 37 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.769973 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 55 FOLLOWER]: Advancing to term 56
I1014 19:25:53.770026 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 55 FOLLOWER]: Advancing to term 56
I1014 19:25:53.770057 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:25:53.770121 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.174s (vote granted)
I1014 19:25:53.771589 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 56.
I1014 19:25:53.771932 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 56 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.772013 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won.
I1014 19:25:53.772069 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 56.
I1014 19:25:53.772256 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:25:53.772358 29621 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 FOLLOWER]: Leader election won for term 56
I1014 19:25:53.772438 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 56 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:53.772549 29621 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.772785 29621 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 54.37, Last appended by leader: 37, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.773813 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.774049 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.865656 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 54 index: 37. Preceding OpId from leader: term: 56 index: 38. (index mismatch)
I1014 19:25:53.866576 32135 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I1014 19:25:53.868131 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.868316 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.869585 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.869827 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.870131 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 56 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 54 index: 37. Preceding OpId from leader: term: 56 index: 38. (index mismatch)
I1014 19:25:53.870018 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.870308 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.870623 32135 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I1014 19:25:53.872079 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.872390 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.872571 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.872769 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:53.972209 32413 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:53.972296 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (starting election)
I1014 19:25:53.972357 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 56 FOLLOWER]: Advancing to term 57
I1014 19:25:53.974448 32413 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.974941 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 57 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:53.975129 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 57 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:53.975795 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 57 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:53.976078 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 57 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.976470 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 57 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:53.976362 32135 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 56 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:53.976632 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 56. Current term is 57. Ops: []
I1014 19:25:53.976681 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 57 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.976662 32135 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:53.977366 32135 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 56.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:53.977614 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 57 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 57 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.977825 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 57 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 57 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:53.977895 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost.
I1014 19:25:53.978106 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:25:53.978181 32413 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I1014 19:25:53.978616 29621 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 LEADER]: Stepping down as leader of term 56
I1014 19:25:53.978703 29621 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:53.978951 29621 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 56.38, Last appended by leader: 38, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:53.979125 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:53.979193 29621 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:53.979333 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 56 FOLLOWER]: Advancing to term 57
I1014 19:25:54.071172 32273 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:54.071267 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:25:54.071362 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 56 FOLLOWER]: Advancing to term 57
I1014 19:25:54.073662 32273 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.074174 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 57 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.074393 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 57 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.074960 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 57 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.075186 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 57: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:54.075639 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 57 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 57: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:54.075796 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 57 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.075980 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (vote granted)
I1014 19:25:54.078073 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 57.
I1014 19:25:54.078523 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 57 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.078610 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 57 election: Election decided. Result: candidate won.
I1014 19:25:54.078900 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:25:54.079039 32273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 FOLLOWER]: Leader election won for term 57
I1014 19:25:54.079483 32273 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.079771 32273 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 56.38, Last appended by leader: 38, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.080953 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 57 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.081248 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:25:54.094102 31406 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.465s	user 0.001s	sys 0.001s
I1014 19:25:54.094183 31406 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:54.095154 31406 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:54.096676 31406 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 49 vs 57: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.097076 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.097347 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.097651 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.097851 32413 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W1014 19:25:54.097954 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.097955 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:25:54.098034 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 57 FOLLOWER]: Advancing to term 58
W1014 19:25:54.098256 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.098618 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.099056 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.099347 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.099603 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.099885 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.100127 31406 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 57 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.100273 32413 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.100792 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 58 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.100980 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 58 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.102061 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 58 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.102291 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 58 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.102854 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 58 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 58 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.103288 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 58 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.103435 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 57 FOLLOWER]: Advancing to term 58
I1014 19:25:54.103555 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (vote granted)
I1014 19:25:54.105918 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 58.
I1014 19:25:54.106261 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 58 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.106320 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won.
I1014 19:25:54.106709 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.151s (election complete)
I1014 19:25:54.106796 32413 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 FOLLOWER]: Leader election won for term 58
I1014 19:25:54.106966 32413 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.107178 32413 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 56.38, Last appended by leader: 38, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.108503 32435 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 58 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.108718 32435 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.109278 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:54.112088 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:54.175464 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 57. Current term is 58. Ops: []
I1014 19:25:54.176105 32443 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.176379 32443 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 58.39 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 38
I1014 19:25:54.176520 32273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 LEADER]: Stepping down as leader of term 57
I1014 19:25:54.176579 32273 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.176771 32273 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 57.39, Last appended by leader: 39, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.176991 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.177050 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:54.177160 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 57 FOLLOWER]: Advancing to term 58
I1014 19:25:54.216212 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:54.216306 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:25:54.216393 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 58 FOLLOWER]: Advancing to term 59
I1014 19:25:54.216703 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 57 index: 39. Preceding OpId from leader: term: 58 index: 39. (term mismatch)
I1014 19:25:54.216775 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 38
I1014 19:25:54.216814 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 57.39
I1014 19:25:54.216888 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:54.217490 32435 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.218277 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.218700 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 59 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.218864 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 59 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.219421 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 59 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.219485 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 59 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.219655 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 59 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.219696 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 59 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.220100 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 59 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 59 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.220273 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 59 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 59 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.220369 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost.
I1014 19:25:54.220309 32435 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 58 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.220584 32435 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.218933 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 58 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.220747 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.220760 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:25:54.220842 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I1014 19:25:54.221192 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 58. Current term is 59. Ops: []
I1014 19:25:54.221225 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 58 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.221413 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.221721 32413 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.222276 32413 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Stepping down as leader of term 58
I1014 19:25:54.222393 32413 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.222609 32413 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 58.39, Last appended by leader: 39, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.222764 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.222822 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:54.222919 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 58 FOLLOWER]: Advancing to term 59
I1014 19:25:54.321838 32273 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:54.321941 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:54.322036 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 58 FOLLOWER]: Advancing to term 59
I1014 19:25:54.323642 32273 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.324076 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 59 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.324287 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 59 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.324990 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 59 candidate_status { last_received { term: 58 index: 39 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.325163 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (vote granted)
I1014 19:25:54.325273 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 59 candidate_status { last_received { term: 58 index: 39 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.325505 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 59: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:54.325860 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 59 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 59: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:54.327225 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 59.
I1014 19:25:54.327538 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 59 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.327606 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 59 election: Election decided. Result: candidate won.
I1014 19:25:54.327833 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:25:54.327914 32273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 FOLLOWER]: Leader election won for term 59
I1014 19:25:54.328043 32273 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.328213 32273 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 58.39, Last appended by leader: 39, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.329138 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 59 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.329365 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.355221 29621 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:54.355315 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.171s (starting election)
I1014 19:25:54.355397 29621 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 59 FOLLOWER]: Advancing to term 60
I1014 19:25:54.356974 29621 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.357452 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 60 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.357635 29621 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 60 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.358409 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 60 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.358502 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 60 candidate_status { last_received { term: 56 index: 38 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.358634 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 60 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.358688 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 59 FOLLOWER]: Advancing to term 60
I1014 19:25:54.359009 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 60 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 60 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.360879 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 60 because replica has last-logged OpId of term: 58 index: 39, which is greater than that of the candidate, which has last-logged OpId of term: 56 index: 38.
I1014 19:25:54.361181 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 60 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 60 because replica has last-logged OpId of term: 58 index: 39, which is greater than that of the candidate, which has last-logged OpId of term: 56 index: 38.
I1014 19:25:54.361260 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost.
I1014 19:25:54.361639 29621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:25:54.361750 29621 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I1014 19:25:54.431627 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 60 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 59. Current term is 60. Ops: []
I1014 19:25:54.432205 32443 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 40, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.432516 32273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 LEADER]: Stepping down as leader of term 59
I1014 19:25:54.432576 32273 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.432803 32273 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 59.40, Last appended by leader: 40, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.432960 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.433035 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:54.433145 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 59 FOLLOWER]: Advancing to term 60
I1014 19:25:54.455627 32413 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:54.455741 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:25:54.455832 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 60 FOLLOWER]: Advancing to term 61
I1014 19:25:54.458091 32413 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.458987 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 61 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.459380 32413 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 61 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.459986 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 61 candidate_status { last_received { term: 58 index: 39 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.460067 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 61 candidate_status { last_received { term: 58 index: 39 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.460181 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 60 FOLLOWER]: Advancing to term 61
I1014 19:25:54.460229 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 60 FOLLOWER]: Advancing to term 61
I1014 19:25:54.460328 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:25:54.462080 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 61 because replica has last-logged OpId of term: 59 index: 40, which is greater than that of the candidate, which has last-logged OpId of term: 58 index: 39.
I1014 19:25:54.462085 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 61.
I1014 19:25:54.462401 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 61 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.462527 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 61 election: Election decided. Result: candidate won.
I1014 19:25:54.462616 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 61 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 61 because replica has last-logged OpId of term: 59 index: 40, which is greater than that of the candidate, which has last-logged OpId of term: 58 index: 39.
I1014 19:25:54.462783 32413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:25:54.462870 32413 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 FOLLOWER]: Leader election won for term 61
I1014 19:25:54.463018 32413 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.463191 32413 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 58.39, Last appended by leader: 39, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.464064 32449 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 61 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.464359 32449 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:25:54.499723 31545 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.622s	user 0.001s	sys 0.000s
I1014 19:25:54.499799 31545 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:54.500490 31545 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:54.501349 31545 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 50 vs 61: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.501582 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.501762 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.501922 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502072 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502238 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502395 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502573 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502724 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.502866 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.503042 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:54.503195 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 61 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.547792 32273 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:54.547895 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:25:54.548004 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 61 FOLLOWER]: Advancing to term 62
I1014 19:25:54.549695 32273 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.550122 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 62 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.550302 32273 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 62 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.550918 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 62 candidate_status { last_received { term: 59 index: 40 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.551082 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 62 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.551415 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 62 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 62 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.551657 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 62 candidate_status { last_received { term: 59 index: 40 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.551846 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 61 FOLLOWER]: Advancing to term 62
I1014 19:25:54.551942 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.176s (vote granted)
I1014 19:25:54.554260 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 62.
I1014 19:25:54.554596 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 62 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.554658 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 62 election: Election decided. Result: candidate won.
I1014 19:25:54.554898 32273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (election complete)
I1014 19:25:54.555006 32273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 FOLLOWER]: Leader election won for term 62
I1014 19:25:54.555124 32273 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.555330 32273 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 59.40, Last appended by leader: 40, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.556213 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.556519 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.556762 31545 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:54.558235 31545 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:54.573727 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 61. Current term is 62. Ops: []
I1014 19:25:54.574265 32449 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 40, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.574573 32413 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 LEADER]: Stepping down as leader of term 61
I1014 19:25:54.574633 32413 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.574874 32413 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 61.40, Last appended by leader: 40, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.575098 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.575171 32413 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:54.575294 32413 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 61 FOLLOWER]: Advancing to term 62
I1014 19:25:54.648639 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 56 index: 38. Preceding OpId from leader: term: 62 index: 41. (index mismatch)
I1014 19:25:54.649153 32443 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 38, Time since last communication: 0.000s
I1014 19:25:54.651274 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.651515 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.651798 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.651721 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.652273 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.652679 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.653178 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.653385 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.653393 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 62 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 61 index: 40. Preceding OpId from leader: term: 62 index: 41. (index mismatch)
I1014 19:25:54.653554 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.653686 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.653898 32443 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 39, Time since last communication: 0.000s
I1014 19:25:54.654000 32443 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 61.40 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 39
I1014 19:25:54.654620 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 39
I1014 19:25:54.654681 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 61.40
I1014 19:25:54.654748 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 62 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:54.655741 32435 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.655767 32449 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.656117 32435 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.656117 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.656370 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.656369 32449 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.755298 32449 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 62 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:54.755425 32449 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:25:54.755498 32449 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 62 FOLLOWER]: Advancing to term 63
I1014 19:25:54.756086 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 62 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.756311 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.757428 32449 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.757789 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 62. Current term is 63. Ops: []
I1014 19:25:54.757980 32449 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 63 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.758182 32449 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 63 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.758357 32467 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 62.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I1014 19:25:54.758724 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 63 candidate_status { last_received { term: 62 index: 41 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.758821 32467 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 LEADER]: Stepping down as leader of term 62
I1014 19:25:54.758929 32467 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:54.759155 32467 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 62.41, Last appended by leader: 41, Current term: 62, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.759369 32467 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.759425 32467 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:54.759660 32467 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 62 FOLLOWER]: Advancing to term 63
I1014 19:25:54.759754 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 63 candidate_status { last_received { term: 62 index: 41 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.759918 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 63 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.760233 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 63 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 63 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.761852 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (vote granted)
I1014 19:25:54.763202 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 63.
I1014 19:25:54.763562 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 63 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.763628 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won.
I1014 19:25:54.763852 32449 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:25:54.763942 32449 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 FOLLOWER]: Leader election won for term 63
I1014 19:25:54.764122 32449 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.764336 32449 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 62.41, Last appended by leader: 41, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.765409 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.765581 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.828176 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:54.828275 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:25:54.828361 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 62 FOLLOWER]: Advancing to term 63
I1014 19:25:54.830054 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.830510 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 63 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.830725 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 63 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:54.831310 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 63 candidate_status { last_received { term: 62 index: 41 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:54.831562 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 63: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:54.831881 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 63 candidate_status { last_received { term: 62 index: 41 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.831967 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 63 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 63: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:54.832098 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 63 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.833292 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 63 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 63 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.833384 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 63 election: Election decided. Result: candidate lost.
I1014 19:25:54.833631 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:25:54.833727 32466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Leader election lost for term 63. Reason: could not achieve majority
I1014 19:25:54.852880 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 62 index: 41. Preceding OpId from leader: term: 63 index: 42. (index mismatch)
I1014 19:25:54.853390 32413 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I1014 19:25:54.854777 32467 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.855007 32467 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.855741 32413 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.855980 32413 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.856216 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 62 index: 41. Preceding OpId from leader: term: 63 index: 42. (index mismatch)
I1014 19:25:54.856321 32467 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.856566 32467 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.856822 32449 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I1014 19:25:54.858417 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.858840 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.858841 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.859136 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:54.937779 32449 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 63 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:54.938030 32449 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:54.956668 32467 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:54.956758 32467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:54.956820 32467 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 63 FOLLOWER]: Advancing to term 64
I1014 19:25:54.958292 32467 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.958890 32467 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 64 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:54.959067 32467 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 64 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:54.959947 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 64 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:54.960162 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 64 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.960592 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 64 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 64 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.960644 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 64 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:54.960871 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 64 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.961215 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 64 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 64 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:54.961290 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost.
I1014 19:25:54.961541 32467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (election complete)
I1014 19:25:54.961634 32467 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I1014 19:25:54.975858 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 63. Current term is 64. Ops: []
I1014 19:25:54.976423 32413 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 63.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I1014 19:25:54.976735 32435 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Stepping down as leader of term 63
I1014 19:25:54.976805 32435 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:54.977036 32435 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 63.42, Last appended by leader: 42, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:54.977213 32435 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:54.977268 32435 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:54.977375 32435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 63 FOLLOWER]: Advancing to term 64
I1014 19:25:55.039109 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:55.039216 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:25:55.039304 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 63 FOLLOWER]: Advancing to term 64
I1014 19:25:55.040931 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.041749 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 64 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.041929 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 64 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.042515 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 64 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:55.042598 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 64 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:55.042680 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (vote granted)
I1014 19:25:55.042770 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 64: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:55.043169 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 64 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 64: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:55.044116 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 64.
I1014 19:25:55.044430 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 64 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.044497 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 64 election: Election decided. Result: candidate won.
I1014 19:25:55.044754 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:25:55.044857 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 FOLLOWER]: Leader election won for term 64
I1014 19:25:55.044965 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:55.045233 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 63.42, Last appended by leader: 42, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.046286 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 64 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.046502 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.046979 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:55.048646 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:55.074693 32467 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:55.074792 32467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.206s (starting election)
I1014 19:25:55.074862 32467 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 64 FOLLOWER]: Advancing to term 65
I1014 19:25:55.076372 32467 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.076825 32467 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 65 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.077433 32467 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 65 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:55.077555 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 65 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:55.077704 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 64 FOLLOWER]: Advancing to term 65
I1014 19:25:55.077775 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (vote granted)
I1014 19:25:55.078003 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 65 candidate_status { last_received { term: 63 index: 42 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:55.078219 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 65 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.078634 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 65 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 65 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.080061 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 65.
I1014 19:25:55.080374 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 65 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.080453 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 65 election: Election decided. Result: candidate won.
I1014 19:25:55.080677 32467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:25:55.080754 32467 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 FOLLOWER]: Leader election won for term 65
I1014 19:25:55.080880 32467 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:55.081090 32467 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 63.42, Last appended by leader: 42, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.081986 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.082234 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.139895 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 64. Current term is 65. Ops: []
I1014 19:25:55.139927 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 64. Current term is 65. Ops: []
I1014 19:25:55.140482 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I1014 19:25:55.141048 32466 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 65.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I1014 19:25:55.141211 29621 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I1014 19:25:55.141412 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 LEADER]: Stepping down as leader of term 64
I1014 19:25:55.141492 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:55.141736 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 64.43, Last appended by leader: 43, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.141945 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:55.142009 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:55.142145 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 64 FOLLOWER]: Advancing to term 65
I1014 19:25:55.188601 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 63 index: 42. Preceding OpId from leader: term: 65 index: 43. (index mismatch)
I1014 19:25:55.189050 32273 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I1014 19:25:55.190110 32435 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.190340 32435 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.191370 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.191642 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.191972 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 65 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 64 index: 43. Preceding OpId from leader: term: 65 index: 43. (term mismatch)
I1014 19:25:55.192047 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 42
I1014 19:25:55.192086 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 64.43
I1014 19:25:55.192142 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 65 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:55.192540 32435 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.192788 32435 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.192875 32467 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I1014 19:25:55.194445 29621 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.194836 29621 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.194445 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.194959 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.275555 32273 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 65 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.275880 32273 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.650354 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 65 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:55.650444 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:55.650544 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 65 FOLLOWER]: Advancing to term 66
I1014 19:25:55.652114 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.652546 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 66 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.652719 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 66 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.653370 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 66 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:55.653368 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 66 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:55.653594 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 66 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.653596 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 66 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.654031 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 66 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 66 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.654183 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 66 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 66 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.654255 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost.
I1014 19:25:55.654503 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:25:55.654579 32466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I1014 19:25:55.664341 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 65. Current term is 66. Ops: []
I1014 19:25:55.664917 32273 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 65.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:55.665168 32273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Stepping down as leader of term 65
I1014 19:25:55.665212 32273 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:55.665392 32273 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 65.43, Last appended by leader: 43, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.665585 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:55.665639 32273 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:55.665756 32273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 65 FOLLOWER]: Advancing to term 66
W1014 19:25:55.709163 31545 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.151s	user 0.001s	sys 0.001s
I1014 19:25:55.709220 31545 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:55.710202 31545 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:55.711318 31545 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 62 vs 66: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.711861 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.712152 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.712468 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.712720 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.712980 31545 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 66 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.733789 32560 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:55.734043 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (starting election)
I1014 19:25:55.734179 32560 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 65 FOLLOWER]: Advancing to term 66
I1014 19:25:55.736326 32560 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.736958 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 66 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.737787 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 66 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:55.737954 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (vote granted)
I1014 19:25:55.740078 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 66 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 66.
I1014 19:25:55.740895 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 66 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.740975 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 66 election: Election decided. Result: candidate won.
I1014 19:25:55.746830 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 66 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:55.747748 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 66 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:55.747925 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:25:55.747995 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 66: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:55.748008 32560 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 FOLLOWER]: Leader election won for term 66
I1014 19:25:55.748129 32560 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:55.748369 32560 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 65.43, Last appended by leader: 43, Current term: 66, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.748675 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 66 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 66: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:55.749449 32560 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 66 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.749758 32560 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.790778 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:55.790904 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (starting election)
I1014 19:25:55.791000 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 66 FOLLOWER]: Advancing to term 67
I1014 19:25:55.793447 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.794067 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 67 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.795100 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 67 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:55.795358 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 67 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.796253 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 67 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 67 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.796558 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 67 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.797282 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 67 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:55.797503 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 66 FOLLOWER]: Advancing to term 67
I1014 19:25:55.797878 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (vote granted)
I1014 19:25:55.800101 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 67.
I1014 19:25:55.800554 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 67 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:55.800632 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won.
I1014 19:25:55.801265 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (election complete)
I1014 19:25:55.801374 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 FOLLOWER]: Leader election won for term 67
I1014 19:25:55.810184 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:55.811647 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 65.43, Last appended by leader: 43, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.813812 32563 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 67 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.814379 32563 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:25:55.822175 32444 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.710s	user 0.001s	sys 0.000s
I1014 19:25:55.822240 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:55.823226 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:55.826153 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 67 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 66. Current term is 67. Ops: []
I1014 19:25:55.826421 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 66. Current term is 67. Ops: []
I1014 19:25:55.826833 32562 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:55.827126 32560 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 LEADER]: Stepping down as leader of term 66
I1014 19:25:55.827244 32560 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:55.827391 32562 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:55.827613 32562 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 67.44 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 43
I1014 19:25:55.827808 32560 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 66.45, Last appended by leader: 45, Current term: 66, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.828066 32560 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:55.828133 32560 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:55.828287 32560 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 66 FOLLOWER]: Advancing to term 67
I1014 19:25:55.910991 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 66 index: 45. Preceding OpId from leader: term: 67 index: 44. (term mismatch)
I1014 19:25:55.911053 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 43
I1014 19:25:55.911085 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 66.44
I1014 19:25:55.911129 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:55.911185 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted WRITE_OP operation due to leader change: 66.45
I1014 19:25:55.911926 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 58 vs 67: Aborted: Transaction aborted by new leader
W1014 19:25:55.912257 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.912250 32563 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W1014 19:25:55.912463 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.912708 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.912988 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.913301 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:55.913925 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 67 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.913679 32560 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 67 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.914052 32560 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.915709 32560 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 67 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.915879 32560 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:55.929544 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 67 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 65 index: 43. Preceding OpId from leader: term: 67 index: 44. (index mismatch)
I1014 19:25:55.932457 32443 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:55.932548 32443 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:25:55.932627 32443 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 67 FOLLOWER]: Advancing to term 68
I1014 19:25:55.933115 32576 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:55.934306 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 67 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:55.934500 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:55.934795 32443 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.935163 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 67. Current term is 68. Ops: []
I1014 19:25:55.935333 32443 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 68 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:55.935530 32443 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 68 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:55.936105 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 68 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:55.936298 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 68 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.936256 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 68 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:55.936455 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 68 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.936638 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 68 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 68 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.936672 32576 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 65.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:55.936852 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 68 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 68 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:55.936923 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost.
I1014 19:25:55.936983 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Stepping down as leader of term 67
I1014 19:25:55.937023 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:55.937153 32443 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:25:55.937238 32443 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I1014 19:25:55.937206 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 67.44, Last appended by leader: 44, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:55.937408 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:55.937475 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:55.937644 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 67 FOLLOWER]: Advancing to term 68
I1014 19:25:56.016719 32560 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:56.016847 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:25:56.016952 32560 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 67 FOLLOWER]: Advancing to term 68
I1014 19:25:56.019016 32560 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.019601 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 68 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.019831 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 68 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.020419 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 68 candidate_status { last_received { term: 67 index: 44 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.020468 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 68 candidate_status { last_received { term: 67 index: 44 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.020658 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 68: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:56.020661 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (vote granted)
I1014 19:25:56.022809 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 68.
I1014 19:25:56.022816 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 68 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 68: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:56.023130 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 68 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.023208 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 68 election: Election decided. Result: candidate won.
I1014 19:25:56.023488 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:25:56.023558 32560 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 FOLLOWER]: Leader election won for term 68
I1014 19:25:56.023691 32560 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:56.023881 32560 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 67.44, Last appended by leader: 44, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.024830 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 68 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.025131 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.025332 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:56.027328 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:56.058593 32443 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:56.058699 32443 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.179s (starting election)
I1014 19:25:56.058782 32443 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 68 FOLLOWER]: Advancing to term 69
I1014 19:25:56.060986 32443 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.061594 32443 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 69 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.061787 32443 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 69 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.062340 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 69 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.062381 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 69 candidate_status { last_received { term: 65 index: 43 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.062569 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 69 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.062556 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 68 FOLLOWER]: Advancing to term 69
I1014 19:25:56.063158 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 69 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 69 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.064739 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 69 because replica has last-logged OpId of term: 67 index: 44, which is greater than that of the candidate, which has last-logged OpId of term: 65 index: 43.
I1014 19:25:56.065186 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 69 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 69 because replica has last-logged OpId of term: 67 index: 44, which is greater than that of the candidate, which has last-logged OpId of term: 65 index: 43.
I1014 19:25:56.065258 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost.
I1014 19:25:56.065503 32443 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.205s (election complete)
I1014 19:25:56.065601 32443 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I1014 19:25:56.101677 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 69 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 68. Current term is 69. Ops: []
I1014 19:25:56.102198 32562 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I1014 19:25:56.102458 32560 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 LEADER]: Stepping down as leader of term 68
I1014 19:25:56.102540 32560 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:56.102774 32560 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 68.45, Last appended by leader: 45, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.102988 32560 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:56.103054 32560 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:56.103173 32560 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 68 FOLLOWER]: Advancing to term 69
I1014 19:25:56.156052 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:56.156160 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:25:56.156253 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 69 FOLLOWER]: Advancing to term 70
I1014 19:25:56.158675 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.159291 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 70 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.159488 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 70 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.160133 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 70 candidate_status { last_received { term: 67 index: 44 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.160168 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 70 candidate_status { last_received { term: 67 index: 44 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.160336 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 69 FOLLOWER]: Advancing to term 70
I1014 19:25:56.160336 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 69 FOLLOWER]: Advancing to term 70
I1014 19:25:56.160428 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (vote granted)
I1014 19:25:56.162590 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 70 because replica has last-logged OpId of term: 68 index: 45, which is greater than that of the candidate, which has last-logged OpId of term: 67 index: 44.
I1014 19:25:56.162580 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 70.
I1014 19:25:56.163023 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 70 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.163103 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won.
I1014 19:25:56.163386 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:25:56.163460 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 70 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 70 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 70 because replica has last-logged OpId of term: 68 index: 45, which is greater than that of the candidate, which has last-logged OpId of term: 67 index: 44.
I1014 19:25:56.163558 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 FOLLOWER]: Leader election won for term 70
I1014 19:25:56.163712 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:56.163920 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 67.44, Last appended by leader: 44, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.165572 32576 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 70 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.165951 32576 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.180644 32560 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 70 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:56.180742 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:25:56.180821 32560 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 70 FOLLOWER]: Advancing to term 71
I1014 19:25:56.182353 32560 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.182843 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 71 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.183019 32560 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 71 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.183678 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 71 candidate_status { last_received { term: 68 index: 45 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.183720 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 71 candidate_status { last_received { term: 68 index: 45 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.183866 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 70 FOLLOWER]: Advancing to term 71
I1014 19:25:56.183895 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 71 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.183957 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:25:56.184371 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 71 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 71 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.186105 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 71.
I1014 19:25:56.186398 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 71 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.186472 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 71 election: Election decided. Result: candidate won.
I1014 19:25:56.186724 32560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:25:56.186796 32560 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 FOLLOWER]: Leader election won for term 71
I1014 19:25:56.186900 32560 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:56.187054 32560 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 68.45, Last appended by leader: 45, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.188248 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.188488 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.244141 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 70. Current term is 71. Ops: []
I1014 19:25:56.244710 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I1014 19:25:56.245033 32466 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 71.46 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
I1014 19:25:56.245165 32576 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 LEADER]: Stepping down as leader of term 70
I1014 19:25:56.245229 32576 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:56.245481 32576 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 70.45, Last appended by leader: 45, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.245651 32576 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:56.245717 32576 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:56.245820 32576 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 70 FOLLOWER]: Advancing to term 71
I1014 19:25:56.285610 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 71 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 65 index: 43. Preceding OpId from leader: term: 71 index: 46. (index mismatch)
I1014 19:25:56.286146 32562 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 43, Time since last communication: 0.000s
I1014 19:25:56.291971 32693 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.292409 32693 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.291903 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.293092 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.293596 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.293814 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.293922 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.294080 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 71 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 70 index: 45. Preceding OpId from leader: term: 71 index: 46. (index mismatch)
I1014 19:25:56.294143 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.294270 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.294562 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.294602 32560 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I1014 19:25:56.294703 32560 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 70.45 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 44
I1014 19:25:56.295419 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 44
I1014 19:25:56.295495 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 70.45
I1014 19:25:56.295557 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 71 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:56.296627 32563 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.296623 32576 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.296871 32563 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.296939 32576 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.297063 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.297199 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.316884 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:56.317597 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:56.319875 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 71 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.320097 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.324525 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:56.326042 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:56.424118 32693 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 71 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:56.424227 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:25:56.424299 32693 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 71 FOLLOWER]: Advancing to term 72
I1014 19:25:56.424295 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 71 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:56.424365 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:25:56.424419 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 71 FOLLOWER]: Advancing to term 72
I1014 19:25:56.426163 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.426462 32693 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.426720 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 72 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.426921 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 72 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.426971 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 72 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.427063 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 72 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.427626 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 72 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.427640 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 72 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.427799 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 72: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:56.427819 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 72: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:56.428063 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 72 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.428295 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 72 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.428431 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 72 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 72: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:56.428117 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 72 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.428648 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 72 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.428730 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 72 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 72: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:56.429052 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 72 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 72 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.429141 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost.
I1014 19:25:56.429157 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 72 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 72 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.429236 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost.
I1014 19:25:56.429373 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (election complete)
I1014 19:25:56.429467 32693 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I1014 19:25:56.429527 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:25:56.429612 32466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I1014 19:25:56.437978 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 71. Current term is 72. Ops: []
I1014 19:25:56.438467 32699 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 71.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I1014 19:25:56.438854 32562 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Stepping down as leader of term 71
I1014 19:25:56.438917 32562 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:56.439149 32562 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 71.47, Last appended by leader: 47, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.439376 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:56.439431 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:56.439589 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 71 FOLLOWER]: Advancing to term 72
I1014 19:25:56.534572 32693 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:56.534689 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (starting election)
I1014 19:25:56.534778 32693 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 72 FOLLOWER]: Advancing to term 73
I1014 19:25:56.537150 32693 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.537758 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 73 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.537962 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 73 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.538591 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 73 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.538767 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 72 FOLLOWER]: Advancing to term 73
I1014 19:25:56.538867 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (vote granted)
I1014 19:25:56.541014 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 73.
I1014 19:25:56.541431 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 73 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.541534 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 73 election: Election decided. Result: candidate won.
I1014 19:25:56.541785 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.204s (election complete)
I1014 19:25:56.541752 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 73 candidate_status { last_received { term: 71 index: 47 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.541872 32693 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 FOLLOWER]: Leader election won for term 73
I1014 19:25:56.541973 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 72 FOLLOWER]: Advancing to term 73
I1014 19:25:56.542057 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.205s (vote granted)
I1014 19:25:56.542084 32693 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:56.542291 32693 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 71.47, Last appended by leader: 47, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.543684 32443 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.543871 32443 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.544160 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 73.
I1014 19:25:56.544554 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 73 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.546042   354 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:56.547632   354 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:25:56.584226 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.536s	user 0.000s	sys 0.001s
I1014 19:25:56.584292 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:56.585178 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:56.586199 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 64 vs 73: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 73 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:56.586452 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 73 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:56.586745 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 73 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:56.586941 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 73 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.638429 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 71 index: 47. Preceding OpId from leader: term: 73 index: 48. (index mismatch)
I1014 19:25:56.639108 32443 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I1014 19:25:56.640403 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.640637 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.642223 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 73 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 71 index: 47. Preceding OpId from leader: term: 73 index: 48. (index mismatch)
I1014 19:25:56.642493 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.642621 32693 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.642884 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.642900 32693 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.643055 32693 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I1014 19:25:56.644546 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.644461 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.644856 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.644912 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:56.722218   355 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 73 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.722641   355 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.824527 32562 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 73 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:56.824681 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:25:56.824751 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 73 FOLLOWER]: Advancing to term 74
I1014 19:25:56.826875 32562 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.827277 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 74 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.827494 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 74 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.828034 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 74 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.828197 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 74 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.828088 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 74 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.828318 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 74 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.828548 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 74 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 74 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.828707 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 74 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 74 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.828779 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 74 election: Election decided. Result: candidate lost.
I1014 19:25:56.829128 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:25:56.829228 32562 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Leader election lost for term 74. Reason: could not achieve majority
I1014 19:25:56.829380 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 73. Current term is 74. Ops: []
I1014 19:25:56.829895   355 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 73.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I1014 19:25:56.830135 32693 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 LEADER]: Stepping down as leader of term 73
I1014 19:25:56.830199 32693 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:56.830423 32693 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 73.48, Last appended by leader: 48, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.830617 32693 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:56.830682 32693 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:56.830830 32693 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 73 FOLLOWER]: Advancing to term 74
I1014 19:25:56.924710 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:56.924811 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:25:56.924888 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 73 FOLLOWER]: Advancing to term 74
I1014 19:25:56.926584 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.927116 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 74 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:56.927284 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 74 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.928040 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 74 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:56.928066 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 74 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.928215 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (vote granted)
I1014 19:25:56.928244 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 74: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:56.929728 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 74 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 74: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:25:56.929855 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 74.
I1014 19:25:56.930199 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 74 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.930268 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 74 election: Election decided. Result: candidate won.
I1014 19:25:56.930634 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (election complete)
I1014 19:25:56.930714 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 FOLLOWER]: Leader election won for term 74
I1014 19:25:56.931102 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:56.931468 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 73.48, Last appended by leader: 48, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.932696   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 74 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.932935   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:56.933148 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:56.934751 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:56.938194 32562 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:56.938287 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (starting election)
I1014 19:25:56.938352 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 74 FOLLOWER]: Advancing to term 75
I1014 19:25:56.939884 32562 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.940263 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 75 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.940464 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 75 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:56.941025 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 75 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:56.941030 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 75 candidate_status { last_received { term: 73 index: 48 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:56.941184 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 75 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.941171 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 74 FOLLOWER]: Advancing to term 75
I1014 19:25:56.941267 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (vote granted)
I1014 19:25:56.941529 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 75 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 75 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:56.943311 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 75.
I1014 19:25:56.943619 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 75 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:56.943691 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 75 election: Election decided. Result: candidate won.
I1014 19:25:56.943904 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.166s (election complete)
I1014 19:25:56.944007 32562 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 FOLLOWER]: Leader election won for term 75
I1014 19:25:56.944087 32562 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:56.944263 32562 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 73.48, Last appended by leader: 48, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:56.945192 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:56.945498 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.026346 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 75 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 74. Current term is 75. Ops: []
I1014 19:25:57.026957   485 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I1014 19:25:57.027254 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 LEADER]: Stepping down as leader of term 74
I1014 19:25:57.027305 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.027493 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 74.49, Last appended by leader: 49, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.027649 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:57.027698 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:57.027791 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 74 FOLLOWER]: Advancing to term 75
I1014 19:25:57.044720 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 75 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 73 index: 48. Preceding OpId from leader: term: 75 index: 49. (index mismatch)
I1014 19:25:57.045298 32699 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I1014 19:25:57.046605 32693 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.046823 32693 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.048388 32693 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.048660 32693 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.048673 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 74 index: 49. Preceding OpId from leader: term: 75 index: 49. (term mismatch)
I1014 19:25:57.048732 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 48
I1014 19:25:57.048776 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 74.49
I1014 19:25:57.048830 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:57.047991 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.049544 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.049546 32699 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I1014 19:25:57.050853   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.051220   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.051057 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.051393 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.144098 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 75 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.144330 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.149193 32693 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 75 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:57.149291 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:25:57.149364 32693 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 75 FOLLOWER]: Advancing to term 76
I1014 19:25:57.151230 32693 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.151664 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 76 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.151872 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 76 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:57.152356 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 76 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.152550 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 76 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.152938 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 76 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 76 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.153059 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 76 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:57.153223 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 76 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.153556 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 76 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 76 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.153637 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 76 election: Election decided. Result: candidate lost.
I1014 19:25:57.153905 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:25:57.154037 32693 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Leader election lost for term 76. Reason: could not achieve majority
I1014 19:25:57.168697 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 75. Current term is 76. Ops: []
I1014 19:25:57.169276 32699 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 75.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I1014 19:25:57.169543 32699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 LEADER]: Stepping down as leader of term 75
I1014 19:25:57.169600 32699 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:57.169833 32699 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 75.49, Last appended by leader: 49, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.169989 32699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:57.170050 32699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:57.170167 32699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 75 FOLLOWER]: Advancing to term 76
I1014 19:25:57.245306 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:25:57.245388 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:25:57.245472 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 75 FOLLOWER]: Advancing to term 76
I1014 19:25:57.247367 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.247938 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 76 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.248106 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 76 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.251125 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 76 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.251125 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 76 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.251358 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 76: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:57.251327 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (vote granted)
I1014 19:25:57.253319 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 76.
I1014 19:25:57.253738 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 76 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 76: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:57.254045 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 76 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.254099 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won.
I1014 19:25:57.254374 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:25:57.254454 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 FOLLOWER]: Leader election won for term 76
I1014 19:25:57.254614 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.254827 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 75.49, Last appended by leader: 49, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.255826   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 76 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.256129   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.279062 32693 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:57.279201 32693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.204s (starting election)
I1014 19:25:57.279278 32693 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 76 FOLLOWER]: Advancing to term 77
I1014 19:25:57.281591 32693 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.282156 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 77 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.283097 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 77 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.283262 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 76 FOLLOWER]: Advancing to term 77
I1014 19:25:57.283349 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (vote granted)
I1014 19:25:57.285593 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 77.
I1014 19:25:57.286188 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 77 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.286258 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 77 election: Election decided. Result: candidate won.
I1014 19:25:57.286634 32693 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 77 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:57.286628   355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.159s (election complete)
I1014 19:25:57.286849   355 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 FOLLOWER]: Leader election won for term 77
I1014 19:25:57.287029   355 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:57.287299   355 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 75.49, Last appended by leader: 49, Current term: 77, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.288311 32693 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.288606 32693 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.289202 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 77 candidate_status { last_received { term: 75 index: 49 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:57.289386 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 77 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.290439 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 77 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 77 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.347440 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 76. Current term is 77. Ops: []
I1014 19:25:57.348062   485 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I1014 19:25:57.348412 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 LEADER]: Stepping down as leader of term 76
I1014 19:25:57.348471 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.348707 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 76.50, Last appended by leader: 50, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.348863 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:57.348930 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:57.349062 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 76 FOLLOWER]: Advancing to term 77
I1014 19:25:57.376529 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 77 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 76 index: 50. Preceding OpId from leader: term: 77 index: 50. (term mismatch)
I1014 19:25:57.376611 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 49
I1014 19:25:57.376654 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 76.50
I1014 19:25:57.376713 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 77 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:57.377424 32693 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I1014 19:25:57.379503 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.379740 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.380686   355 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.380946   355 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.381073 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.381217 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.381871 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 75 index: 49. Preceding OpId from leader: term: 77 index: 50. (index mismatch)
I1014 19:25:57.382378 32693 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I1014 19:25:57.383945 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.384162 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.384390 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.384577 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:57.469015   355 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.469314   355 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:25:57.498634 32444 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.173s	user 0.002s	sys 0.000s
I1014 19:25:57.498690 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:57.499392 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:57.500277 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 71 vs 77: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.500546 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.500744 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.500962 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.501133 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.501320 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 77 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:57.554191   354 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.006s	user 0.001s	sys 0.000s
I1014 19:25:57.554250   354 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:57.554955   354 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:57.561156   354 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:57.562949   354 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:57.661290 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 77 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:57.661387 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:25:57.661500 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 77 FOLLOWER]: Advancing to term 78
I1014 19:25:57.663735 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.664290 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 78 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.665072 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 78 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.665277 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 78 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.665279 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 78 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.665696 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 78 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 78 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.665997 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 78 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.666196 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 78 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.666544 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 78 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 78 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.666620 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost.
I1014 19:25:57.666867 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:25:57.666990 32466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I1014 19:25:57.679857 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 77. Current term is 78. Ops: []
I1014 19:25:57.680382   355 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 77.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:57.680702   610 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 LEADER]: Stepping down as leader of term 77
I1014 19:25:57.680774   610 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:57.681004   610 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 77.51, Last appended by leader: 51, Current term: 77, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.681185   610 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:57.681246   610 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:57.681346   610 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 77 FOLLOWER]: Advancing to term 78
I1014 19:25:57.745206 32562 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:57.745308 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:25:57.745383 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 77 FOLLOWER]: Advancing to term 78
I1014 19:25:57.746886 32562 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.747303 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 78 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.747534 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 78 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:57.748057 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 78 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.748250 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (vote granted)
I1014 19:25:57.748242 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 78 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:57.748450 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 78: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:57.748893 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 78 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 78: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:57.749989 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 78.
I1014 19:25:57.750319 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 78 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.750394 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 78 election: Election decided. Result: candidate won.
I1014 19:25:57.750643 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:25:57.750746 32562 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 FOLLOWER]: Leader election won for term 78
I1014 19:25:57.750867 32562 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:57.751088 32562 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 77.51, Last appended by leader: 51, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.751965 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 78 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.752187 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.752408 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:57.753938 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:57.782891 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:57.782984 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.168s (starting election)
I1014 19:25:57.783079 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 78 FOLLOWER]: Advancing to term 79
I1014 19:25:57.784735 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.785154 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 79 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.785336 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 79 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.785902 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 79 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.786131 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 79 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.786219 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 79 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.786376 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 78 FOLLOWER]: Advancing to term 79
I1014 19:25:57.786468 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (vote granted)
I1014 19:25:57.786643 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 79 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 79 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.789185 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 79.
I1014 19:25:57.789536 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 79 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.789598 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 79 election: Election decided. Result: candidate won.
I1014 19:25:57.789979 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.178s (election complete)
I1014 19:25:57.790051 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 FOLLOWER]: Leader election won for term 79
I1014 19:25:57.790161 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.790357 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 77.51, Last appended by leader: 51, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.791227   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 79 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.791522   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.843500 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 79 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 78. Current term is 79. Ops: []
I1014 19:25:57.844056 32699 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:57.844313 32699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 LEADER]: Stepping down as leader of term 78
I1014 19:25:57.844372 32699 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:57.844595 32699 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 78.52, Last appended by leader: 52, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.844748 32699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:57.844810 32699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:57.844923 32699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 78 FOLLOWER]: Advancing to term 79
I1014 19:25:57.908267   610 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:57.908356   610 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:25:57.908453   610 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 79 FOLLOWER]: Advancing to term 80
I1014 19:25:57.910050   610 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.910661   610 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 80 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.910851   610 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 80 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:57.911201 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 80 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 79. Current term is 80. Ops: []
I1014 19:25:57.911397 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 80 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:57.911422 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 80 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.911624 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 80 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.911605 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 79 FOLLOWER]: Advancing to term 80
I1014 19:25:57.912050 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 80 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 80 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.912225   485 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:57.912452 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 LEADER]: Stepping down as leader of term 79
I1014 19:25:57.912544 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.912804 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 79.52, Last appended by leader: 52, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.913010 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:57.913071 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:57.913192 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 79 FOLLOWER]: Advancing to term 80
I1014 19:25:57.913517 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 80 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 80 because replica has last-logged OpId of term: 78 index: 52, which is greater than that of the candidate, which has last-logged OpId of term: 77 index: 51.
I1014 19:25:57.914083 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 80 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 80 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 80 because replica has last-logged OpId of term: 78 index: 52, which is greater than that of the candidate, which has last-logged OpId of term: 77 index: 51.
I1014 19:25:57.914165 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 80 election: Election decided. Result: candidate lost.
I1014 19:25:57.914479   610 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:25:57.914556   610 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 80 FOLLOWER]: Leader election lost for term 80. Reason: could not achieve majority
I1014 19:25:57.937706 32699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 80 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:57.937815 32699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:25:57.937892 32699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 80 FOLLOWER]: Advancing to term 81
I1014 19:25:57.939864 32699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.940346 32699 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 81 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.940603 32699 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 81 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:57.941121 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 81 candidate_status { last_received { term: 78 index: 52 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.941310 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 80 FOLLOWER]: Advancing to term 81
I1014 19:25:57.941411 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.195s (vote granted)
I1014 19:25:57.943445 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 81.
I1014 19:25:57.943913 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 81 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.944036 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 81 election: Election decided. Result: candidate won.
I1014 19:25:57.944314 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (election complete)
I1014 19:25:57.944399 32562 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 FOLLOWER]: Leader election won for term 81
I1014 19:25:57.944386 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 81 candidate_status { last_received { term: 78 index: 52 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:57.944563 32562 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:57.944609 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 80 FOLLOWER]: Advancing to term 81
I1014 19:25:57.944774 32562 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 78.52, Last appended by leader: 52, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.945796 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 81 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.946115 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:57.946827 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 81 because replica has last-logged OpId of term: 79 index: 52, which is greater than that of the candidate, which has last-logged OpId of term: 78 index: 52.
I1014 19:25:57.947238 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 81 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 81 because replica has last-logged OpId of term: 79 index: 52, which is greater than that of the candidate, which has last-logged OpId of term: 78 index: 52.
I1014 19:25:57.990973 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:57.991060 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:25:57.991139 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 81 FOLLOWER]: Advancing to term 82
I1014 19:25:57.992691 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.993139 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 82 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:57.993345 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 82 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.994042 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 82 candidate_status { last_received { term: 79 index: 52 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:57.994124 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 82 candidate_status { last_received { term: 79 index: 52 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:57.994241 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 82 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.994298 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 81 FOLLOWER]: Advancing to term 82
I1014 19:25:57.994431 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (vote granted)
I1014 19:25:57.994832 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 82 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 82 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:57.996098 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 82.
I1014 19:25:57.996407 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 82 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:57.996476 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won.
I1014 19:25:57.996722 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:25:57.996800 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 FOLLOWER]: Leader election won for term 82
I1014 19:25:57.996951 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:57.997259 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 79.52, Last appended by leader: 52, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:57.998193   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 82 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:57.998456   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.044215 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 81. Current term is 82. Ops: []
I1014 19:25:58.044780 32699 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:58.045017 32699 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 82.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I1014 19:25:58.045141 32562 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 LEADER]: Stepping down as leader of term 81
I1014 19:25:58.045202 32562 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:58.045444 32562 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 81.53, Last appended by leader: 53, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.045591 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:58.045655 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.045778 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 81 FOLLOWER]: Advancing to term 82
I1014 19:25:58.101228   610 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 82 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:58.101318   610 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.176s (starting election)
I1014 19:25:58.101399   610 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 82 FOLLOWER]: Advancing to term 83
I1014 19:25:58.102927   610 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.103335   610 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 83 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.103502   610 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 83 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.104152 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 83 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.104223 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 83 candidate_status { last_received { term: 77 index: 51 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.104300 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 82 FOLLOWER]: Advancing to term 83
I1014 19:25:58.104444 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 83 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.105243 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 83 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 83 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.106060 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 83 because replica has last-logged OpId of term: 81 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 77 index: 51.
I1014 19:25:58.106258 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 83 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 82. Current term is 83. Ops: []
I1014 19:25:58.106528 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 83 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 83 because replica has last-logged OpId of term: 81 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 77 index: 51.
I1014 19:25:58.106606 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 83 election: Election decided. Result: candidate lost.
I1014 19:25:58.106843   610 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.182s (election complete)
I1014 19:25:58.106914   610 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 83 FOLLOWER]: Leader election lost for term 83. Reason: could not achieve majority
I1014 19:25:58.107168   485 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:58.107370   485 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 81.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I1014 19:25:58.107542 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Stepping down as leader of term 82
I1014 19:25:58.107615 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:58.107884 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 82.53, Last appended by leader: 53, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.108098 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.108186 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:58.108330 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 82 FOLLOWER]: Advancing to term 83
I1014 19:25:58.122984 32562 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 83 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:58.123085 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:58.123149 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 83 FOLLOWER]: Advancing to term 84
I1014 19:25:58.125056 32562 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.125483 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 84 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.125694 32562 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 84 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.126210 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 84 candidate_status { last_received { term: 81 index: 53 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:58.126379 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 83 FOLLOWER]: Advancing to term 84
I1014 19:25:58.126471 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.288s (vote granted)
I1014 19:25:58.126647 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 84 candidate_status { last_received { term: 81 index: 53 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.126824 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 83 FOLLOWER]: Advancing to term 84
I1014 19:25:58.127938 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 84.
I1014 19:25:58.128242 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 84 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.128315 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 84 election: Election decided. Result: candidate won.
I1014 19:25:58.128528 32562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (election complete)
I1014 19:25:58.128609 32562 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 FOLLOWER]: Leader election won for term 84
I1014 19:25:58.128751 32562 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:58.128904 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 84 because replica has last-logged OpId of term: 82 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 81 index: 53.
I1014 19:25:58.128968 32562 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 81.53, Last appended by leader: 53, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.129230 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 84 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 84 because replica has last-logged OpId of term: 82 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 81 index: 53.
I1014 19:25:58.130134 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 84 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.130328 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.223212 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 84 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:58.223301 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:25:58.223381 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 84 FOLLOWER]: Advancing to term 85
I1014 19:25:58.224895 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.225277 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 85 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.225469 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 85 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.226047 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 85 candidate_status { last_received { term: 82 index: 53 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.226047 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 85 candidate_status { last_received { term: 82 index: 53 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:58.226207 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 84 FOLLOWER]: Advancing to term 85
I1014 19:25:58.226238 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 85 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.226294 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.164s (vote granted)
I1014 19:25:58.227728 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 85 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 85 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.227797 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 85.
I1014 19:25:58.228109 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 85 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.228170 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 85 election: Election decided. Result: candidate won.
I1014 19:25:58.228404 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:25:58.228500 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 FOLLOWER]: Leader election won for term 85
I1014 19:25:58.228654 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:58.228889 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 82.53, Last appended by leader: 53, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.230015   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.230252   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.246752 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 84. Current term is 85. Ops: []
I1014 19:25:58.247334 32699 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:58.247622 32699 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 85.54 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I1014 19:25:58.247763 32562 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 LEADER]: Stepping down as leader of term 84
I1014 19:25:58.247829 32562 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:58.248030 32562 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 84.54, Last appended by leader: 54, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.248208 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:58.248260 32562 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.248384 32562 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 84 FOLLOWER]: Advancing to term 85
I1014 19:25:58.309993 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 85 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 77 index: 51. Preceding OpId from leader: term: 85 index: 54. (index mismatch)
I1014 19:25:58.310881   485 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:58.312454   610 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.312714   610 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.314486   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.314780   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.315117 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 84 index: 54. Preceding OpId from leader: term: 85 index: 54. (term mismatch)
I1014 19:25:58.315187 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 53
I1014 19:25:58.315233 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 84.54
I1014 19:25:58.315294 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:58.316390   999 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 51, Time since last communication: 0.000s
I1014 19:25:58.316655   999 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 81.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I1014 19:25:58.317575 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 51
I1014 19:25:58.317620 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 78.52
I1014 19:25:58.317683 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:58.317421   610 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.317754 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 81.53
I1014 19:25:58.317770   610 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.317811 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:58.317502  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.318032  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.318212  1000 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.318801  1000 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.320266 32562 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.320751 32699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.321297 32699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.321293 32562 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.320350  1005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.321712  1005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.320638  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.321858  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.409060 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 85 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.409406 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.415024  1000 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 85 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:58.415110  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:25:58.415185  1000 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 85 FOLLOWER]: Advancing to term 86
I1014 19:25:58.416735  1000 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.417137  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 86 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.417345  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 86 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.417946 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 86 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.417939 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 86 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.418121 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 86 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.418128 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 86 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.418573 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 86 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 86 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.418766 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 86 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 86 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.418850 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 86 election: Election decided. Result: candidate lost.
I1014 19:25:58.419061  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:25:58.419134  1000 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 86 FOLLOWER]: Leader election lost for term 86. Reason: could not achieve majority
I1014 19:25:58.419514 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 86 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 85. Current term is 86. Ops: []
I1014 19:25:58.419968 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 85.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I1014 19:25:58.420219   999 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Stepping down as leader of term 85
I1014 19:25:58.420277   999 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:58.420527   999 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 85.54, Last appended by leader: 54, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.420748   999 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.420810   999 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:58.420943   999 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 85 FOLLOWER]: Advancing to term 86
I1014 19:25:58.506564   999 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:58.506662   999 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (starting election)
I1014 19:25:58.506737   999 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 86 FOLLOWER]: Advancing to term 87
I1014 19:25:58.508942   999 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.509425   999 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 87 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.509665   999 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 87 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.510205  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:58.510365  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:25:58.510360 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 87 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.510493  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 85 FOLLOWER]: Advancing to term 86
I1014 19:25:58.511072 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 87 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:58.511241 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 86 FOLLOWER]: Advancing to term 87
I1014 19:25:58.511322 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.164s (vote granted)
I1014 19:25:58.512789  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.513273  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 86 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.513419 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 87.
I1014 19:25:58.513468  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 86 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.513713 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 86 FOLLOWER]: Advancing to term 87
I1014 19:25:58.513785 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (vote granted)
I1014 19:25:58.513921 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 86 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:58.514098 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 86. Current term is 87.
I1014 19:25:58.514349 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 86 candidate_status { last_received { term: 85 index: 54 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.514561 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 86. Current term is 87.
W1014 19:25:58.514554 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 86 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 86. Current term is 87.
I1014 19:25:58.514631 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 86 election: Cancelling election due to peer responding with higher term
W1014 19:25:58.515036 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 86 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 86. Current term is 87.
I1014 19:25:58.515311 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 87 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.515377 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 87 election: Election decided. Result: candidate won.
I1014 19:25:58.515647   999 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (election complete)
I1014 19:25:58.515740   999 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 FOLLOWER]: Leader election won for term 87
I1014 19:25:58.515903   999 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:58.516119 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 87.
I1014 19:25:58.516269  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (election complete)
I1014 19:25:58.516206   999 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 85.54, Last appended by leader: 54, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.516434  1004 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Leader election lost for term 86. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 86. Current term is 87.
I1014 19:25:58.516623 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 87 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.517267 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.517562 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.623404 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 85 index: 54. Preceding OpId from leader: term: 87 index: 55. (index mismatch)
I1014 19:25:58.624675 32466 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I1014 19:25:58.626044  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.626344  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.627048   999 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.627281   999 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.628135 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 85 index: 54. Preceding OpId from leader: term: 87 index: 55. (index mismatch)
I1014 19:25:58.628358  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.628589  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.628638   999 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I1014 19:25:58.630050  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.630280  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.630332  1000 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.630657  1000 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.730221  1000 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:58.730319  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (starting election)
I1014 19:25:58.730404  1000 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 87 FOLLOWER]: Advancing to term 88
I1014 19:25:58.732645  1000 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.733177  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 88 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.733698  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 88 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.734066 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 88 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.734248 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 88 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.734553 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 88 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.734684 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 88 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 88 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.734753 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 88 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.735083 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 88 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 88 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.735172 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 88 election: Election decided. Result: candidate lost.
I1014 19:25:58.735406  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:25:58.735527  1000 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Leader election lost for term 88. Reason: could not achieve majority
I1014 19:25:58.749343 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 87 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.749596 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.749822 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 87. Current term is 88. Ops: []
I1014 19:25:58.750283 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 87.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I1014 19:25:58.750514   485 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 LEADER]: Stepping down as leader of term 87
I1014 19:25:58.750569   485 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:58.750813   485 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 87.55, Last appended by leader: 55, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.750957   485 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.750999   485 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:58.751102   485 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 87 FOLLOWER]: Advancing to term 88
I1014 19:25:58.826990  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:58.827080  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:25:58.827157  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 87 FOLLOWER]: Advancing to term 88
I1014 19:25:58.828589  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.828974  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 88 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:58.829136  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 88 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.829619 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 88 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:58.829712 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 88 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.829813 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 88: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:58.829908 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:25:58.830159 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 88 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 88: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:58.831567 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 88.
I1014 19:25:58.831854 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 88 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.831923 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 88 election: Election decided. Result: candidate won.
I1014 19:25:58.832162  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:25:58.832237  1004 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 FOLLOWER]: Leader election won for term 88
I1014 19:25:58.832657  1004 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:58.832871  1004 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 87.55, Last appended by leader: 55, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.833853  1141 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 88 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.834076  1141 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.871804  1000 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:58.871892  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (starting election)
I1014 19:25:58.871966  1000 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 88 FOLLOWER]: Advancing to term 89
I1014 19:25:58.874089  1000 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.874581  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 89 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:58.874964  1000 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 89 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.875283 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 89 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:58.875500 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 89 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.875564 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 89 candidate_status { last_received { term: 87 index: 55 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:58.875710 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 88 FOLLOWER]: Advancing to term 89
I1014 19:25:58.875803 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:25:58.875871 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 89 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 89 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:58.877288 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 89.
I1014 19:25:58.877612 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 89 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:58.877686 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 89 election: Election decided. Result: candidate won.
I1014 19:25:58.877913  1000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:25:58.877984  1000 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 FOLLOWER]: Leader election won for term 89
I1014 19:25:58.878134  1000 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:58.878429  1000 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 87.55, Last appended by leader: 55, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.879698  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.879906  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.911579 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 89 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 88. Current term is 89. Ops: []
I1014 19:25:58.912310  1141 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I1014 19:25:58.912581  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 LEADER]: Stepping down as leader of term 88
I1014 19:25:58.912637  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:58.912734 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 88. Current term is 89. Ops: []
I1014 19:25:58.912816  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 88.56, Last appended by leader: 56, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:58.913050  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:58.913229  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:58.913352  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 88 FOLLOWER]: Advancing to term 89
W1014 19:25:58.929318 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.995s	user 0.001s	sys 0.000s
I1014 19:25:58.929376 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:58.930132 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:58.930971 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 74 vs 89: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.931185 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.931362 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.931586 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.931742 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.931916 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.932106 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.932301 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.932566 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:58.932760 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 89 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.967075 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 88 index: 56. Preceding OpId from leader: term: 89 index: 56. (term mismatch)
I1014 19:25:58.967152 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 55
I1014 19:25:58.967195 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 88.56
I1014 19:25:58.967249 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:58.968056  1002 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I1014 19:25:58.969763  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.969956  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.970837  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.971084  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:58.971333 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 89 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 87 index: 55. Preceding OpId from leader: term: 89 index: 56. (index mismatch)
I1014 19:25:58.971997  1000 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I1014 19:25:58.972385  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.972622  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.973515 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.973583   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:58.973836   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:58.973836 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.049903  1000 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.050179  1000 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.241719 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 89 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:59.241827 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:25:59.241902 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 89 FOLLOWER]: Advancing to term 90
I1014 19:25:59.244160 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.244706 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 90 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.244927 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 90 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.245528 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 90 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.245739 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 90 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.246376 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 90 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.246529 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 90 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 90 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.246609 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 90 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.246991 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 90 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 90 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.247066 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 90 election: Election decided. Result: candidate lost.
I1014 19:25:59.247319 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:25:59.247475 32466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Leader election lost for term 90. Reason: could not achieve majority
I1014 19:25:59.264044 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 89. Current term is 90. Ops: []
I1014 19:25:59.264695  1002 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 89.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.264997  1002 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Stepping down as leader of term 89
I1014 19:25:59.265058  1002 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:25:59.265326  1002 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 89.56, Last appended by leader: 56, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.265563  1002 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:59.265628  1002 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:59.265771  1002 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 89 FOLLOWER]: Advancing to term 90
W1014 19:25:59.310987 32444 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.557s	user 0.002s	sys 0.000s
I1014 19:25:59.311059 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:59.311970 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:25:59.313704 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 78 vs 89: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.314074 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.314339 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.314541 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 89 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.330561  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:25:59.330678  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (starting election)
I1014 19:25:59.330762  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 89 FOLLOWER]: Advancing to term 90
I1014 19:25:59.332443  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.332854  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 90 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.333012  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 90 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:59.333560 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 90 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.333639 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 90 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:59.333721 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (vote granted)
I1014 19:25:59.333853 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 90: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:59.334252 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 90 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 90: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:25:59.335218 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 90.
I1014 19:25:59.335530 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 90 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.335592 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 90 election: Election decided. Result: candidate won.
I1014 19:25:59.335819  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:25:59.335883  1004 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 FOLLOWER]: Leader election won for term 90
I1014 19:25:59.335994  1004 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:59.336172  1004 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 89.56, Last appended by leader: 56, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.336931  1141 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 90 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.337227  1141 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.337436 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:59.338912 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:59.375849 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:59.376032 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:25:59.376195 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 90 FOLLOWER]: Advancing to term 91
I1014 19:25:59.378701 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.379348 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 91 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.379608 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 91 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.380064 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 91 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.380266 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 91 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.380491 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 91 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.380641 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 90 FOLLOWER]: Advancing to term 91
I1014 19:25:59.380633 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 91 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 91 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.380731 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (vote granted)
I1014 19:25:59.382169 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 91.
I1014 19:25:59.382477 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 91 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.382556 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 91 election: Election decided. Result: candidate won.
I1014 19:25:59.382843 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.201s (election complete)
I1014 19:25:59.382927 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 FOLLOWER]: Leader election won for term 91
I1014 19:25:59.383113 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:59.383329 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 89.56, Last appended by leader: 56, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.384934   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 91 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.385087   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.385278 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:59.386916 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:59.416781 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 90. Current term is 91. Ops: []
I1014 19:25:59.417367  1141 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.417610  1141 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 91.57 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 56
I1014 19:25:59.417695  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 LEADER]: Stepping down as leader of term 90
I1014 19:25:59.417754  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:25:59.417982  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 90.57, Last appended by leader: 57, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.418112  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:25:59.418155  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:59.418242  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 90 FOLLOWER]: Advancing to term 91
W1014 19:25:59.451645   354 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.889s	user 0.002s	sys 0.000s
I1014 19:25:59.451715   354 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:59.452446   354 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:25:59.453267   354 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 77 vs 91: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.453512   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.453702   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.453876   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.454051   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.454217   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.454392   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.454608   354 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 91 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.482774 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 90 index: 57. Preceding OpId from leader: term: 91 index: 57. (term mismatch)
I1014 19:25:59.482846 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 56
I1014 19:25:59.482897 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 90.57
I1014 19:25:59.482980 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:25:59.483793   485 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.485049  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 91 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.485271  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.486656   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 91 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.486873   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.487231  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 91 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.487479  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.488054 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 91 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 89 index: 56. Preceding OpId from leader: term: 91 index: 57. (index mismatch)
I1014 19:25:59.488513 32466 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.488971  1259 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 91 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:59.489135  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:25:59.489284  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 91 FOLLOWER]: Advancing to term 92
I1014 19:25:59.490803  1259 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.491117 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 91. Current term is 92. Ops: []
I1014 19:25:59.491320  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 92 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.491614  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 92 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:59.491719 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 89.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.491953 32466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 LEADER]: Stepping down as leader of term 91
I1014 19:25:59.492009 32466 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:59.492020 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 92 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.492202 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 92 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.492195 32466 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 91.57, Last appended by leader: 57, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.492470 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:59.492527 32466 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:59.492575 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 92 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 92 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.492676 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 91 FOLLOWER]: Advancing to term 92
I1014 19:25:59.493125 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 92 candidate_status { last_received { term: 89 index: 56 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:59.495599 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 92 because replica has last-logged OpId of term: 91 index: 57, which is greater than that of the candidate, which has last-logged OpId of term: 89 index: 56.
I1014 19:25:59.496091 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 92 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 92 because replica has last-logged OpId of term: 91 index: 57, which is greater than that of the candidate, which has last-logged OpId of term: 89 index: 56.
I1014 19:25:59.496186 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost.
I1014 19:25:59.496552  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:25:59.496619  1259 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I1014 19:25:59.587265  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:59.587366  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:25:59.587448  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 91 FOLLOWER]: Advancing to term 92
I1014 19:25:59.587826 32466 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:59.587915 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:25:59.587978 32466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 92 FOLLOWER]: Advancing to term 93
I1014 19:25:59.589020  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.589411  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 92 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.589648  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 92 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:59.590322 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 92 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.590209 32466 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.590422 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 92 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:59.590555 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 92: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:59.590659 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 92. Current term is 93.
I1014 19:25:59.590857 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 93 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.591110 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 92 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 92: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:25:59.591174 32466 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 93 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:25:59.591280 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 92 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 92. Current term is 93.
I1014 19:25:59.591321 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 92 election: Cancelling election due to peer responding with higher term
I1014 19:25:59.591636 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 93 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.591766 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 92 FOLLOWER]: Advancing to term 93
I1014 19:25:59.591845 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (vote granted)
I1014 19:25:59.591884  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:25:59.591987  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 92 FOLLOWER]: Advancing to term 93
I1014 19:25:59.593978  1004 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 93 FOLLOWER]: Leader election lost for term 92. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 92. Current term is 93.
I1014 19:25:59.594174 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 93 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.594317 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (vote granted)
I1014 19:25:59.594597 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 93.
I1014 19:25:59.595844 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 93 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.595932 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won.
I1014 19:25:59.595947 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 93.
I1014 19:25:59.596181 32466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (election complete)
I1014 19:25:59.596369 32466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Leader election won for term 93
I1014 19:25:59.596377 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 93 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.596657 32466 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:59.596894 32466 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 91.57, Last appended by leader: 57, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.597965   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 93 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.598120   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.698240 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 89 index: 56. Preceding OpId from leader: term: 93 index: 58. (index mismatch)
I1014 19:25:59.698556  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 93 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:59.698648  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:25:59.698722  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 93 FOLLOWER]: Advancing to term 94
I1014 19:25:59.698909   485 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s
I1014 19:25:59.700906  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.701422  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 94 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.702064  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 94 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:59.702073 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 94 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.702719 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 94 candidate_status { last_received { term: 91 index: 57 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:59.702889 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 94 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.703191 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 94 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 94 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.703774  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 93 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.704195  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.704035  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 93 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.704304  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.704403 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 94 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:59.704731 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 93 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.704991 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.705199 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 94 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 94 because replica is already servicing an update from a current leader or another vote.
I1014 19:25:59.705284 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost.
I1014 19:25:59.705587  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.155s (election complete)
I1014 19:25:59.705739  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 93 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.705958  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.706037  1004 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I1014 19:25:59.706159 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 94 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 93. Current term is 94. Ops: []
I1014 19:25:59.706712  1520 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I1014 19:25:59.706955   485 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 LEADER]: Stepping down as leader of term 93
I1014 19:25:59.707007   485 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:59.707223   485 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 93.58, Last appended by leader: 58, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.707366   485 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:25:59.707458   485 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:25:59.707587   485 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 93 FOLLOWER]: Advancing to term 94
I1014 19:25:59.783730 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:25:59.784523 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:25:59.785523 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 91 vs 94: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 94 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.785825 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 94 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.786036 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 94 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:25:59.786298 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 94 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.793864   485 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:25:59.793936   485 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:25:59.793999   485 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 94 FOLLOWER]: Advancing to term 95
I1014 19:25:59.795629   485 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.796030   485 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 95 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.796206   485 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 95 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:25:59.796942 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 95 candidate_status { last_received { term: 93 index: 58 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:25:59.796916 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 95 candidate_status { last_received { term: 93 index: 58 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.797096 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 94 FOLLOWER]: Advancing to term 95
I1014 19:25:59.797154 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 95 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.797183 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.256s (vote granted)
I1014 19:25:59.797523 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 95 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 95 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.799294 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 95.
I1014 19:25:59.799631 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 95 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.799700 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 95 election: Election decided. Result: candidate won.
I1014 19:25:59.799947   485 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:25:59.800019   485 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 FOLLOWER]: Leader election won for term 95
I1014 19:25:59.800169   485 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:25:59.800374   485 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 93.58, Last appended by leader: 58, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.801523  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.801697  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.801910 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:25:59.803400 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:25:59.805639  1259 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:25:59.805722  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:25:59.805810  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 93 FOLLOWER]: Advancing to term 94
I1014 19:25:59.808100  1259 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:25:59.808569  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 94 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:25:59.808744  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 94 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:25:59.809375 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 94 candidate_status { last_received { term: 93 index: 58 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:25:59.809558 28639 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 94. Current term is 95.
W1014 19:25:59.809903 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 94 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 94. Current term is 95.
I1014 19:25:59.809947 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 94 election: Cancelling election due to peer responding with higher term
I1014 19:25:59.810120  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:25:59.810233  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 94 FOLLOWER]: Advancing to term 95
I1014 19:25:59.810907 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 94 candidate_status { last_received { term: 93 index: 58 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:25:59.811133 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 94 because replica is either leader or believes a valid leader to be alive.
W1014 19:25:59.811691 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 94 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 94 because replica is either leader or believes a valid leader to be alive.
I1014 19:25:59.812260  1259 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 95 FOLLOWER]: Leader election lost for term 94. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 94. Current term is 95.
I1014 19:25:59.911648 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 95 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 93 index: 58. Preceding OpId from leader: term: 95 index: 59. (index mismatch)
I1014 19:25:59.912164  1520 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I1014 19:25:59.913570  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.913818  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.914558   485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.914834   485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:25:59.915192  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.915395  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.915527 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 91 index: 57. Preceding OpId from leader: term: 95 index: 59. (index mismatch)
I1014 19:25:59.915992   485 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I1014 19:25:59.917887  1141 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.917949  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.918176  1141 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.918272  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:25:59.918404  1141 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:25:59.918633  1141 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.008512 32466 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 95 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.008761 32466 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.015131  1259 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 95 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:00.015228  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (starting election)
I1014 19:26:00.015295  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 95 FOLLOWER]: Advancing to term 96
I1014 19:26:00.016906  1259 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.017313  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 96 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.017537  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 96 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.018071 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 96 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.018214 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 96 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.018414 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 96 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.018518 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 96 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.018959 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 96 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 96 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.019131 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 96 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 96 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.019204 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost.
I1014 19:26:00.019404  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:00.019528  1259 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
I1014 19:26:00.020695 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 95. Current term is 96. Ops: []
I1014 19:26:00.021178 32466 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 95.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I1014 19:26:00.021448  1520 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Stepping down as leader of term 95
I1014 19:26:00.021503  1520 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:00.021750  1520 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 95.59, Last appended by leader: 59, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.021922  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:00.021987  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:00.022089  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 95 FOLLOWER]: Advancing to term 96
I1014 19:26:00.110213  1634 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:00.110383  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:00.110500  1634 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 95 FOLLOWER]: Advancing to term 96
I1014 19:26:00.112921  1634 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.114073  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 96 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.114507  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 96 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.114892 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 96 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.115105 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 96: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:00.115444 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 96 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.115650 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (vote granted)
I1014 19:26:00.117864 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 96.
I1014 19:26:00.118252 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 96 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.118335 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 96 election: Election decided. Result: candidate won.
I1014 19:26:00.118619  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:00.118687 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 96 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 96: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:00.118760  1634 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 FOLLOWER]: Leader election won for term 96
I1014 19:26:00.118957  1634 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.119184  1634 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 95.59, Last appended by leader: 59, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.120322  1141 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 96 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.120540  1141 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.163748  1259 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.163863  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.182s (starting election)
I1014 19:26:00.163941  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 96 FOLLOWER]: Advancing to term 97
I1014 19:26:00.166113  1259 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.166685  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 97 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.167064  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 97 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.167403 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 97 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.167665 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 97 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.167693 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 97 candidate_status { last_received { term: 95 index: 59 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.167881 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 96 FOLLOWER]: Advancing to term 97
I1014 19:26:00.167953 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:00.168069 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 97 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 97 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.169600 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 97.
I1014 19:26:00.169894 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 97 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.169956 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 97 election: Election decided. Result: candidate won.
I1014 19:26:00.170174  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.204s (election complete)
I1014 19:26:00.170281  1259 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 FOLLOWER]: Leader election won for term 97
I1014 19:26:00.170418  1259 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:00.170636  1259 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 95.59, Last appended by leader: 59, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.171494  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.171741  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.172236  1758 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:00.173895  1758 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:00.213961 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:00.214970 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:00.217324 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 96. Current term is 97. Ops: []
I1014 19:26:00.217324 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 97 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 96. Current term is 97. Ops: []
I1014 19:26:00.217943  1634 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I1014 19:26:00.218145  1634 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 97.60 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 59
I1014 19:26:00.218271  1141 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I1014 19:26:00.218528  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 LEADER]: Stepping down as leader of term 96
I1014 19:26:00.218590  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.218804  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 96.61, Last appended by leader: 61, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.218981  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:00.219048  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:00.219171  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 96 FOLLOWER]: Advancing to term 97
I1014 19:26:00.264329 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 97 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 95 index: 59. Preceding OpId from leader: term: 97 index: 60. (index mismatch)
I1014 19:26:00.264854  1002 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I1014 19:26:00.266180  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.266506  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.268155 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 97 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 96 index: 61. Preceding OpId from leader: term: 97 index: 60. (term mismatch)
I1014 19:26:00.268218 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 59
I1014 19:26:00.268105  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.268246 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 96.60
I1014 19:26:00.267851  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.268290  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.268293 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 97 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:00.268309  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.268344 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted WRITE_OP operation due to leader change: 96.61
I1014 19:26:00.268852 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 90 vs 97: Aborted: Transaction aborted by new leader
W1014 19:26:00.269114 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 97 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.269110  1002 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I1014 19:26:00.270611  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.270514  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.271075  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.271153  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.350035  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 97 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.350436  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.451714  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 97 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:00.451850  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:00.451936  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 97 FOLLOWER]: Advancing to term 98
I1014 19:26:00.454274  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.454859  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 98 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.455094  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 98 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.455659 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 98 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.455844 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 98 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.455788 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 98 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.455971 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 97 FOLLOWER]: Advancing to term 98
I1014 19:26:00.456107 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (vote granted)
I1014 19:26:00.456260 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 98 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 98 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.457645 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 98.
I1014 19:26:00.457793  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 98 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.457892  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:00.457942 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 98 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.458000 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 98 election: Election decided. Result: candidate won.
I1014 19:26:00.458003  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 98 FOLLOWER]: Advancing to term 99
I1014 19:26:00.458197  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:00.458298  1004 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 FOLLOWER]: Leader election won for term 98
I1014 19:26:00.458416  1004 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.458640  1004 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 97.60, Last appended by leader: 60, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.459540  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 98 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.459796  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.459992 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:00.459985  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.460525  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 99 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.460736  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 99 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.461366 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 99 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.461613 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 99 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.462039 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 99 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 99 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.462086 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:00.462182 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 99 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.462335 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 99 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.462646 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 99 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 99 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.462709 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost.
I1014 19:26:00.462939  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:00.463078  1520 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I1014 19:26:00.466313 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 97. Current term is 98. Ops: []
I1014 19:26:00.467057  1848 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 97.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I1014 19:26:00.467356  1848 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 98.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I1014 19:26:00.467458 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 97. Current term is 99. Ops: []
I1014 19:26:00.467491  1848 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Stepping down as leader of term 97
I1014 19:26:00.467542  1848 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:00.467744  1848 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 97.60, Last appended by leader: 60, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.467972  1848 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:00.468039  1848 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:00.468152  1848 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 97 FOLLOWER]: Advancing to term 98
I1014 19:26:00.547677 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 98. Current term is 99. Ops: []
I1014 19:26:00.548224  1634 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I1014 19:26:00.548545  1634 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Stepping down as leader of term 98
I1014 19:26:00.548604  1634 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.548799  1634 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 98.61, Last appended by leader: 61, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.548970  1848 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 98 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.549052  1634 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:00.549052  1848 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:00.549108  1634 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:00.549124  1848 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 98 FOLLOWER]: Advancing to term 99
I1014 19:26:00.549245  1634 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 98 FOLLOWER]: Advancing to term 99
I1014 19:26:00.551625  1848 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.552011  1848 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 99 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.552197  1848 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 99 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.552626 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 99 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.552856 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 99 because replica has last-logged OpId of term: 98 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 97 index: 60.
I1014 19:26:00.553056 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 99 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.553252 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 99: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:00.553329 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 99 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 99 because replica has last-logged OpId of term: 98 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 97 index: 60.
I1014 19:26:00.553633 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 99 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 99: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:00.553712 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost.
I1014 19:26:00.553936  1848 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:00.554044  1848 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I1014 19:26:00.587275  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.587371  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:00.587478  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 99 FOLLOWER]: Advancing to term 100
I1014 19:26:00.589679  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.590262  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 100 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.590538  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 100 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.591099 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 100 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.591161 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 100 candidate_status { last_received { term: 97 index: 60 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.591274 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 99 FOLLOWER]: Advancing to term 100
I1014 19:26:00.591339 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 99 FOLLOWER]: Advancing to term 100
I1014 19:26:00.591437 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.179s (vote granted)
I1014 19:26:00.592834 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 100 because replica has last-logged OpId of term: 98 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 97 index: 60.
I1014 19:26:00.593194 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 100 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 100 because replica has last-logged OpId of term: 98 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 97 index: 60.
I1014 19:26:00.593524 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 100.
I1014 19:26:00.593837 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 100 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.593930 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won.
I1014 19:26:00.594214  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.220s (election complete)
I1014 19:26:00.594287  1520 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 FOLLOWER]: Leader election won for term 100
I1014 19:26:00.594745  1520 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:00.595000  1520 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 97.60, Last appended by leader: 60, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.596138  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 100 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.596451  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.660219  1634 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 100 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.660322  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:00.660398  1634 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 100 FOLLOWER]: Advancing to term 101
I1014 19:26:00.662019  1634 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.662488  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 101 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.662722  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 101 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.663358 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 101 candidate_status { last_received { term: 98 index: 61 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.663329 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 101 candidate_status { last_received { term: 98 index: 61 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.663513 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 100 FOLLOWER]: Advancing to term 101
I1014 19:26:00.663563 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 101 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.663614 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:26:00.663920 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 101 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 101 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.665663 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 101 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 101.
I1014 19:26:00.666060 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 101 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.666139 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 101 election: Election decided. Result: candidate won.
I1014 19:26:00.666366  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:00.666450  1634 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 FOLLOWER]: Leader election won for term 101
I1014 19:26:00.666617  1634 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.666852  1634 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 98.61, Last appended by leader: 61, Current term: 101, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.668128  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.668296  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.687299 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 101 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 100. Current term is 101. Ops: []
I1014 19:26:00.687891  1912 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I1014 19:26:00.688215  1520 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 LEADER]: Stepping down as leader of term 100
I1014 19:26:00.688285  1520 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:00.688589  1520 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 100.61, Last appended by leader: 61, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.688810  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:00.688887  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:00.689035  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 100 FOLLOWER]: Advancing to term 101
I1014 19:26:00.748025 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 100 index: 61. Preceding OpId from leader: term: 101 index: 62. (index mismatch)
I1014 19:26:00.748562  1004 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I1014 19:26:00.748770  1004 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 100.61 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 60
I1014 19:26:00.749454 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 60
I1014 19:26:00.749500 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 100.61
I1014 19:26:00.749578 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:00.750236  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.750649  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.751401  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.751643  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.752102 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 101 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 97 index: 60. Preceding OpId from leader: term: 101 index: 62. (index mismatch)
I1014 19:26:00.752527  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.752524  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.752898  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.752988  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.753118  1634 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I1014 19:26:00.754940  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.755321  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.755317  1848 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.755497  1848 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.755029  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.755641  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:00.781452  1758 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:00.782279  1758 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:00.783246  1758 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 97 vs 101: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:00.783485  1758 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:00.783725  1758 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.854871  1002 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 101 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:00.854964  1002 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:00.855039  1002 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 101 FOLLOWER]: Advancing to term 102
I1014 19:26:00.857259  1002 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.857776  1002 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 102 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.857952  1002 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 102 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.858619 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 102 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.858824 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 102 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.859390 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 102 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.859645 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 102 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.859539  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 101 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.859779  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.859808 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 101. Current term is 102. Ops: []
I1014 19:26:00.860132 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 102 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 102 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.860249 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 102 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 102 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.860229  1634 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 101.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I1014 19:26:00.860335 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 102 election: Election decided. Result: candidate lost.
I1014 19:26:00.860448  1634 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 LEADER]: Stepping down as leader of term 101
I1014 19:26:00.860549  1634 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:00.860674  1002 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:00.860827  1002 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Leader election lost for term 102. Reason: could not achieve majority
I1014 19:26:00.860780  1634 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 101.62, Last appended by leader: 62, Current term: 101, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.861045  1634 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:00.861102  1634 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:00.861256  1634 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 101 FOLLOWER]: Advancing to term 102
I1014 19:26:00.945827  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:00.945931  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:00.946024  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 101 FOLLOWER]: Advancing to term 102
I1014 19:26:00.948247  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.948873  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 102 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.949123  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 102 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:00.949784 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 102 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.949962 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (vote granted)
I1014 19:26:00.950084 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 102 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:00.950280 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 102: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:00.950754 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 102 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 102: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:00.951612 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 102.
I1014 19:26:00.951928 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 102 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.951990 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won.
I1014 19:26:00.952241  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:00.952322  1520 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 FOLLOWER]: Leader election won for term 102
I1014 19:26:00.952442  1520 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:00.952775  1520 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 101.62, Last appended by leader: 62, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.953941  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 102 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.954294  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.978215  1002 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:00.978310  1002 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.209s (starting election)
I1014 19:26:00.978415  1002 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 102 FOLLOWER]: Advancing to term 103
I1014 19:26:00.980413  1002 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.980974  1002 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 103 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.981134  1002 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 103 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:00.981942 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 103 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:00.982182 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 103 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.982132 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 103 candidate_status { last_received { term: 101 index: 62 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:00.982298 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 102 FOLLOWER]: Advancing to term 103
I1014 19:26:00.982414 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:00.983974 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 103.
I1014 19:26:00.984318 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 103 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:00.984424 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 103 election: Election decided. Result: candidate won.
I1014 19:26:00.984751 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 103 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 103 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:00.985033  1002 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.160s (election complete)
I1014 19:26:00.985133  1002 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 FOLLOWER]: Leader election won for term 103
I1014 19:26:00.985260  1002 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:00.985499  1002 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 101.62, Last appended by leader: 62, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:00.986902  1002 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:00.987100  1002 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:00.987287  1758 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:00.989200  1758 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:01.062399 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 102. Current term is 103. Ops: []
I1014 19:26:01.063009  1912 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I1014 19:26:01.063305  1520 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 LEADER]: Stepping down as leader of term 102
I1014 19:26:01.063360  1520 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:01.063612  1520 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 102.63, Last appended by leader: 63, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.063766  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:01.063835  1520 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:01.063946  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 102 FOLLOWER]: Advancing to term 103
I1014 19:26:01.073009 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 103 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 102 index: 63. Preceding OpId from leader: term: 103 index: 63. (term mismatch)
I1014 19:26:01.073073 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 62
I1014 19:26:01.073115 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 102.63
I1014 19:26:01.073181 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 103 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:01.073694  1002 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I1014 19:26:01.075086  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.075282  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.076309  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.076592  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.076745 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 101 index: 62. Preceding OpId from leader: term: 103 index: 63. (index mismatch)
I1014 19:26:01.077486  1259 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I1014 19:26:01.077472  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.077664  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.078996  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.079206  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.079460  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.079617  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.152806  1758 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:01.153565  1758 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:01.155553  1848 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 103 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.155771  1848 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.259739  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 103 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:01.259835  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:01.259922  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 103 FOLLOWER]: Advancing to term 104
I1014 19:26:01.261518  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.261974  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 104 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:01.262177  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 104 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.262941 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 104 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:01.263136 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 104 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.263128 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 104 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:01.263355 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 104 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.263664 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 104 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 104 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.263846 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 104 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 104 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.263922 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 104 election: Election decided. Result: candidate lost.
I1014 19:26:01.264348 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 103. Current term is 104. Ops: []
I1014 19:26:01.264541  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:01.264653  1520 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Leader election lost for term 104. Reason: could not achieve majority
I1014 19:26:01.265020  1848 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 103.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I1014 19:26:01.265246  1848 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 LEADER]: Stepping down as leader of term 103
I1014 19:26:01.265306  1848 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:01.265501  1848 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 103.64, Last appended by leader: 64, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.265736  1848 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:01.265800  1848 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:01.265923  1848 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 103 FOLLOWER]: Advancing to term 104
I1014 19:26:01.341768  1634 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:01.341893  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:01.341976  1634 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 103 FOLLOWER]: Advancing to term 104
I1014 19:26:01.343504  1634 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.343933  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 104 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.344135  1634 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 104 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.344774 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 104 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:01.344826 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 104 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:01.344959 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:26:01.344996 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 104: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:01.345321 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 104 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 104: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:01.347062 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 104.
I1014 19:26:01.347375 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 104 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.347456 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won.
I1014 19:26:01.347720  1634 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:01.347828  1634 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 FOLLOWER]: Leader election won for term 104
I1014 19:26:01.347965  1634 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:01.348193  1634 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 103.64, Last appended by leader: 64, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.349081  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 104 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.349351  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.411085  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:01.411203  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.150s (starting election)
I1014 19:26:01.411286  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 104 FOLLOWER]: Advancing to term 105
I1014 19:26:01.412904  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.413311  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 105 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:01.413542  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 105 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.414070 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 105 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:01.414296 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 105 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.414459 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 105 candidate_status { last_received { term: 103 index: 64 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:01.414628 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 104 FOLLOWER]: Advancing to term 105
I1014 19:26:01.414660 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 105 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 105 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.414713 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:01.416852 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 105.
I1014 19:26:01.417214 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 105 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.417276 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 105 election: Election decided. Result: candidate won.
I1014 19:26:01.417521  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:01.417590  1520 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 FOLLOWER]: Leader election won for term 105
I1014 19:26:01.417718  1520 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:01.417902  1520 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 103.64, Last appended by leader: 64, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.418695  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.418956  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.438061 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 104. Current term is 105. Ops: []
I1014 19:26:01.438781  1004 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I1014 19:26:01.439066  1004 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 105.65 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 64
I1014 19:26:01.439173  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 LEADER]: Stepping down as leader of term 104
I1014 19:26:01.439236  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:01.439425  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 104.65, Last appended by leader: 65, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.439643  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:01.439700  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:01.439826  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 104 FOLLOWER]: Advancing to term 105
I1014 19:26:01.501178 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 103 index: 64. Preceding OpId from leader: term: 105 index: 65. (index mismatch)
I1014 19:26:01.501863  1520 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I1014 19:26:01.503283  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.503536  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.504757 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 105 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 104 index: 65. Preceding OpId from leader: term: 105 index: 65. (term mismatch)
I1014 19:26:01.504829 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 64
I1014 19:26:01.504863 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 104.65
I1014 19:26:01.504709  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.504936  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.504951 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 105 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:01.506078  1520 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I1014 19:26:01.506220  1259 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.506391  1259 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.507853  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.508129  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.508317  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.508466  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:01.579466 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.776s	user 0.002s	sys 0.000s
I1014 19:26:01.579556 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:01.580642 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:01.582528  1912 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 105 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.582738  1912 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.585919 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:01.588652 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:01.688433  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 105 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:01.688524  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:01.688594  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 105 FOLLOWER]: Advancing to term 106
I1014 19:26:01.690177  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.690668  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 106 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.690896  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 106 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.691661 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 105. Current term is 106. Ops: []
I1014 19:26:01.691838 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 106 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:01.692131 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 106 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.692315 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 106 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:01.692494 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 106 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.692844 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 106 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 106 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.692221  2389 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 105.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.001s
I1014 19:26:01.693042 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 106 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 106 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.693130 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 106 election: Election decided. Result: candidate lost.
I1014 19:26:01.693125  2389 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Stepping down as leader of term 105
I1014 19:26:01.693195  2389 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:01.693460  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:01.693548  1004 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Leader election lost for term 106. Reason: could not achieve majority
I1014 19:26:01.693401  2389 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 105.66, Last appended by leader: 66, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.693711  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:01.693773  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:01.693898  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 105 FOLLOWER]: Advancing to term 106
I1014 19:26:01.777225  1259 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:01.777324  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:01.777415  1259 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 105 FOLLOWER]: Advancing to term 106
I1014 19:26:01.779686  1259 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.780181  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 106 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:01.780438  1259 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 106 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.780977 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 106 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:01.781196 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 106: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:01.781152 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 106 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:01.781332 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
I1014 19:26:01.781679 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 106 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 106: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:01.783078 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 106.
I1014 19:26:01.783411 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 106 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.783475 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 106 election: Election decided. Result: candidate won.
I1014 19:26:01.783742  1259 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:01.783833  1259 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 FOLLOWER]: Leader election won for term 106
I1014 19:26:01.784210  1259 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:01.784520  1259 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 105.66, Last appended by leader: 66, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.785862  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 106 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.786044  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.786612  2502 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:01.788607  2502 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:01.840960  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:01.841091  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:01.841162  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 106 FOLLOWER]: Advancing to term 107
I1014 19:26:01.843487  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.843991  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 107 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:01.844185  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 107 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.844920 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 107 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:01.845113 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 107 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.845105 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 107 candidate_status { last_received { term: 105 index: 66 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:01.845265 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 106 FOLLOWER]: Advancing to term 107
I1014 19:26:01.845358 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:01.845651 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 107 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 107 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:01.847528 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 107.
I1014 19:26:01.848796 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 107 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:01.848873 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 107 election: Election decided. Result: candidate won.
I1014 19:26:01.849118  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.219s (election complete)
I1014 19:26:01.849200  1004 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 FOLLOWER]: Leader election won for term 107
I1014 19:26:01.849447  1004 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:01.849824  1004 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 105.66, Last appended by leader: 66, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.851143  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.851358  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.865800 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 107 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 106. Current term is 107. Ops: []
I1014 19:26:01.866457  2501 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I1014 19:26:01.866745  2501 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 LEADER]: Stepping down as leader of term 106
I1014 19:26:01.866819  2501 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:01.867058  2501 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 106.67, Last appended by leader: 67, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:01.867300  2501 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:01.867367  2501 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:01.867547  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 106 FOLLOWER]: Advancing to term 107
I1014 19:26:01.935986 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 107 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 106 index: 67. Preceding OpId from leader: term: 107 index: 67. (term mismatch)
I1014 19:26:01.936053 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 66
I1014 19:26:01.936097 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 106.67
I1014 19:26:01.936151 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 107 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:01.936909  1004 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I1014 19:26:01.938105  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.938480  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.939774  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.940085  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.940466  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.940752  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:01.942306 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 107 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 105 index: 66. Preceding OpId from leader: term: 107 index: 67. (index mismatch)
I1014 19:26:01.942795  1004 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I1014 19:26:01.944254  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.944344  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 107 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:01.944583  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:01.944645  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:02.038925 32444 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.577s	user 0.002s	sys 0.000s
I1014 19:26:02.038986 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:02.039775  2501 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 107 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:02.039917  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:02.040006 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:02.040020  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 107 FOLLOWER]: Advancing to term 108
I1014 19:26:02.041949  2501 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.042289 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 107. Current term is 108. Ops: []
I1014 19:26:02.042418  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 108 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.042636  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 108 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.043000  1004 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 107.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I1014 19:26:02.043277 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 108 candidate_status { last_received { term: 107 index: 67 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.043638  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 LEADER]: Stepping down as leader of term 107
I1014 19:26:02.043685  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:02.043932  2389 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 107 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:02.044036  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:02.043920  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 107.68, Last appended by leader: 68, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.044137  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 107 FOLLOWER]: Advancing to term 108
I1014 19:26:02.044253 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 108 candidate_status { last_received { term: 107 index: 67 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.045955  2389 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.046392  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 108 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.046550 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 108: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:02.046633  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 108 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.047117  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:02.047183  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:02.047317  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 107 FOLLOWER]: Advancing to term 108
I1014 19:26:02.047715 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 108 candidate_status { last_received { term: 107 index: 67 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.047750 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 108 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 108: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:02.047972 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 108 candidate_status { last_received { term: 107 index: 67 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:02.048154 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 108: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:02.048494 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 108 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 108: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:02.049412 28639 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 108 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:02.049659 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 108 because replica has last-logged OpId of term: 107 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 107 index: 67.
I1014 19:26:02.049968 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 108 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 108 because replica has last-logged OpId of term: 107 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 107 index: 67.
I1014 19:26:02.050071 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 108 election: Election decided. Result: candidate lost.
I1014 19:26:02.050312  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:02.050390  2389 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 108 FOLLOWER]: Leader election lost for term 108. Reason: could not achieve majority
I1014 19:26:02.050477 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 108 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 108 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:02.050585 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 108 election: Election decided. Result: candidate lost.
I1014 19:26:02.050796  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:02.050880  2501 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Leader election lost for term 108. Reason: could not achieve majority
I1014 19:26:02.159121  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:02.159220  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:02.159312  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 108 FOLLOWER]: Advancing to term 109
I1014 19:26:02.160858  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.161300  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 109 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.161512  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 109 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.162147 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 109 candidate_status { last_received { term: 107 index: 68 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:02.162225 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 109 candidate_status { last_received { term: 107 index: 68 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.162331 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 108 FOLLOWER]: Advancing to term 109
I1014 19:26:02.162386 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 108 FOLLOWER]: Advancing to term 109
I1014 19:26:02.162446 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.189s (vote granted)
I1014 19:26:02.162506 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.207s (vote granted)
I1014 19:26:02.163964 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 109.
I1014 19:26:02.164264 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 109 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.164324 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 109 election: Election decided. Result: candidate won.
I1014 19:26:02.164481 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 109.
I1014 19:26:02.164532  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:02.164587  1004 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 FOLLOWER]: Leader election won for term 109
I1014 19:26:02.164682  1004 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:02.164971 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 109 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.164971  1004 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 107.68, Last appended by leader: 68, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.165923  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.166251  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.261404 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 109 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 107 index: 67. Preceding OpId from leader: term: 109 index: 69. (index mismatch)
I1014 19:26:02.261934  2516 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I1014 19:26:02.263559  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.263856  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.265522  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.265717  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.266149 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:02.266125 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 109 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 107 index: 67. Preceding OpId from leader: term: 109 index: 69. (index mismatch)
I1014 19:26:02.266856  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.267127  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.267395  1634 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 67, Time since last communication: 0.000s
I1014 19:26:02.268319 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:02.270522  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.270612  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.270910  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.271025  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.366142  2501 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 109 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:02.366263  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:02.366343  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 109 FOLLOWER]: Advancing to term 110
I1014 19:26:02.368245  2501 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.368674  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 110 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.368880  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 110 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.369539  1520 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 109 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:02.369627  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:02.369624 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 110 candidate_status { last_received { term: 109 index: 69 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.369714  1520 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 109 FOLLOWER]: Advancing to term 110
I1014 19:26:02.369825 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 110 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.370720  1634 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 109 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.370923  1634 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.371927  1520 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.372349 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 109. Current term is 110. Ops: []
I1014 19:26:02.372437  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 110 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.372681 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 109. Current term is 110. Ops: []
I1014 19:26:02.372680 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 110 candidate_status { last_received { term: 109 index: 69 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.372921 29010 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 110 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:02.373312 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 110 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 110 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.373486 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 110 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Service unavailable: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 110 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:02.373569 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 110 election: Election decided. Result: candidate lost.
I1014 19:26:02.373764  1520 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 110 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.374125  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:02.374095  1004 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 109.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I1014 19:26:02.374202  2501 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Leader election lost for term 110. Reason: could not achieve majority
I1014 19:26:02.374351  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 LEADER]: Stepping down as leader of term 109
I1014 19:26:02.374460  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:02.374789 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 110 candidate_status { last_received { term: 109 index: 69 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.374747  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 109.69, Last appended by leader: 69, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.375170  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:02.375229  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:02.375347  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 109 FOLLOWER]: Advancing to term 110
I1014 19:26:02.375336 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 110 candidate_status { last_received { term: 109 index: 69 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:02.375545 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 110: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:02.375892 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 110 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 110: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:02.377107 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
W1014 19:26:02.377470  2516 consensus_queue.cc:1042] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4043e04130264237805f012d8e251bd4" responder_term: 110 status { last_received { term: 109 index: 69 } last_committed_idx: 69 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 109. Current term is 110. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I1014 19:26:02.378918 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 110.
I1014 19:26:02.379309 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 110 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.379375 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 110 election: Election decided. Result: candidate won.
I1014 19:26:02.379596  1520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:02.379663  1520 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Leader election won for term 110
I1014 19:26:02.379812  1520 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:02.380013  1520 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 109.69, Last appended by leader: 69, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.381152  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.381381  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.469322 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 110 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 109 index: 69. Preceding OpId from leader: term: 110 index: 70. (index mismatch)
I1014 19:26:02.469880  1520 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I1014 19:26:02.472116  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.472352  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.473877  1520 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.474099  1520 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.475126 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 109 index: 69. Preceding OpId from leader: term: 110 index: 70. (index mismatch)
I1014 19:26:02.475724  2389 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I1014 19:26:02.475878  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.476078  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.478019  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.478461  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.478255  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.478811  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.563819  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 110 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.564218  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.576930  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 110 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:02.577014  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:02.577096  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 110 FOLLOWER]: Advancing to term 111
I1014 19:26:02.578852  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.579298  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 111 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.579471  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 111 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.580310 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 111 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.580550 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 111 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.580853 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 110. Current term is 111. Ops: []
I1014 19:26:02.580854 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 111 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:02.581044 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 111 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.581298 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 111 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 111 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.581725  2389 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 110.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I1014 19:26:02.581943  2780 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 LEADER]: Stepping down as leader of term 110
I1014 19:26:02.581987 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 111 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 111 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.582015  2780 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:02.582088 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 111 election: Election decided. Result: candidate lost.
I1014 19:26:02.582319  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:02.582393  2516 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Leader election lost for term 111. Reason: could not achieve majority
I1014 19:26:02.582242  2780 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 110.70, Last appended by leader: 70, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.582454  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:02.582550  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:02.582685  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 110 FOLLOWER]: Advancing to term 111
I1014 19:26:02.664643  2776 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:02.664752  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:02.664829  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 110 FOLLOWER]: Advancing to term 111
I1014 19:26:02.666496  2776 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.666908  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 111 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.667093  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 111 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.667629 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 111 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.667680 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 111 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.667832 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 111: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:02.667853 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (vote granted)
I1014 19:26:02.668308 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 111 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 111: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:02.669284 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 111.
I1014 19:26:02.669589 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 111 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.669667 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 111 election: Election decided. Result: candidate won.
I1014 19:26:02.669898  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:02.669975  2776 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 FOLLOWER]: Leader election won for term 111
I1014 19:26:02.670133  2776 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:02.670325  2776 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 110.70, Last appended by leader: 70, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.671353  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 111 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.671520  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.730613  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:02.730715  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:02.730794  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 111 FOLLOWER]: Advancing to term 112
I1014 19:26:02.732851  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.733387  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 112 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:02.733582  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 112 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.734133 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 112 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:02.734334 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 112 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.734403 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 112 candidate_status { last_received { term: 110 index: 70 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.734616 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 111 FOLLOWER]: Advancing to term 112
I1014 19:26:02.734709 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (vote granted)
I1014 19:26:02.734823 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 112 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 112 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.736896 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 112.
I1014 19:26:02.737316 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 112 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.737381 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 112 election: Election decided. Result: candidate won.
I1014 19:26:02.737749  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.203s (election complete)
I1014 19:26:02.737844  2516 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 FOLLOWER]: Leader election won for term 112
I1014 19:26:02.738076  2516 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:02.738463  2516 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 110.70, Last appended by leader: 70, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.741271  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.741472  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.757706 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 111. Current term is 112. Ops: []
I1014 19:26:02.758216  2501 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I1014 19:26:02.758503  2501 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 LEADER]: Stepping down as leader of term 111
I1014 19:26:02.758561  2501 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:02.758798  2501 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 111.71, Last appended by leader: 71, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.758941  2501 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:02.759006  2501 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:02.759096  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 111 FOLLOWER]: Advancing to term 112
I1014 19:26:02.786686  2502 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:02.787643  2502 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:02.788707  2502 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 106 vs 112: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 112 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:02.788987  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 112 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.842039 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 110 index: 70. Preceding OpId from leader: term: 112 index: 71. (index mismatch)
I1014 19:26:02.842650  2516 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I1014 19:26:02.843772  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.843960  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.844944  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.845146  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.845597 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 112 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 111 index: 71. Preceding OpId from leader: term: 112 index: 71. (term mismatch)
I1014 19:26:02.845660 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 70
I1014 19:26:02.845705 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 111.71
I1014 19:26:02.845625  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.845764  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.845753 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 112 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:02.846298  2516 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I1014 19:26:02.847729  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.847960  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.848168  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.848346  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:02.948380  2501 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 112 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:02.948474  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:02.948588  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 112 FOLLOWER]: Advancing to term 113
I1014 19:26:02.950793  2501 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.951305  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 113 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:02.951561  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 113 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:02.951958 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 113 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:02.952160 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 113 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.952347 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 113 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:02.952548 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 113 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 113 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.952576 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 113 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.952868 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 113 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 113 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:02.952940 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 113 election: Election decided. Result: candidate lost.
I1014 19:26:02.953152  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:02.953224  2501 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Leader election lost for term 113. Reason: could not achieve majority
I1014 19:26:02.966197  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 112 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:02.966480  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:02.966629 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 112. Current term is 113. Ops: []
I1014 19:26:02.967173  1004 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 112.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I1014 19:26:02.967403  1004 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 LEADER]: Stepping down as leader of term 112
I1014 19:26:02.967458  1004 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:02.967694  1004 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 112.71, Last appended by leader: 71, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:02.967842  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:02.967901  1004 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:02.968008  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 112 FOLLOWER]: Advancing to term 113
I1014 19:26:03.040865  2780 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:03.040964  2780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:03.041049  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 112 FOLLOWER]: Advancing to term 113
I1014 19:26:03.042819  2780 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.043287  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 113 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.043453  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 113 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.044004 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 113 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.044178 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:03.044136 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 113 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.044348 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 113: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:03.044845 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 113 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 113: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:03.046010 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 113.
I1014 19:26:03.046310 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 113 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.046383 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 113 election: Election decided. Result: candidate won.
I1014 19:26:03.046656  2780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:03.046730  2780 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 FOLLOWER]: Leader election won for term 113
I1014 19:26:03.046828  2780 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:03.047030  2780 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 112.71, Last appended by leader: 71, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.048050  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 113 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.048321  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.060617  2501 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:03.060714  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.179s (starting election)
I1014 19:26:03.060803  2501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 113 FOLLOWER]: Advancing to term 114
I1014 19:26:03.063015  2501 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.063604  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 114 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.063813  2501 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 114 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.064612 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 114 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.064777 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 113 FOLLOWER]: Advancing to term 114
I1014 19:26:03.064862 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:03.067010 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 114.
I1014 19:26:03.067451 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 114 candidate_status { last_received { term: 112 index: 71 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.067776 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 114 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.068205 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 114 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 114 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.068378 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 114 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.068454 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 114 election: Election decided. Result: candidate won.
I1014 19:26:03.068871  2501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:03.068979  2501 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 FOLLOWER]: Leader election won for term 114
I1014 19:26:03.069159  2501 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:03.069428  2501 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 112.71, Last appended by leader: 71, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.070963  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.071111  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.071317  2502 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:03.073340  2502 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:03.154470 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 113. Current term is 114. Ops: []
I1014 19:26:03.155089  2389 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I1014 19:26:03.155339  2389 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 114.72 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 71
I1014 19:26:03.155448  2780 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 LEADER]: Stepping down as leader of term 113
I1014 19:26:03.155504  2780 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:03.155740  2780 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 113.72, Last appended by leader: 72, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.155897  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:03.155980  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:03.156102  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 113 FOLLOWER]: Advancing to term 114
I1014 19:26:03.171052 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 114 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 112 index: 71. Preceding OpId from leader: term: 114 index: 72. (index mismatch)
I1014 19:26:03.171726  2501 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I1014 19:26:03.172216 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 114 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 113 index: 72. Preceding OpId from leader: term: 114 index: 72. (term mismatch)
I1014 19:26:03.172284 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 71
I1014 19:26:03.172334 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 113.72
I1014 19:26:03.172396 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 114 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:03.173018  2501 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I1014 19:26:03.173235  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.173509  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.174408  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.174960  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.174803  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.175185  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.175439  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.175664  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.176224  2501 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.176506  2501 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.176488  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.176700  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.190245 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:03.191275 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:03.192349 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 109 vs 114: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.192677 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.192920 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.193186 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.193511 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.193787 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.194023 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.194288 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.194562 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.194813 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.195127 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:03.195389 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 114 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.275730  1004 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 114 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:03.275830  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:03.275916  1004 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 114 FOLLOWER]: Advancing to term 115
I1014 19:26:03.276674  2780 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 114 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:03.276785  2780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:03.276883  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 114 FOLLOWER]: Advancing to term 115
I1014 19:26:03.278048  1004 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.278594  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 115 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.279006  1004 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 115 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.279026  2780 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.279440 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 115 candidate_status { last_received { term: 114 index: 72 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.279678 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 115 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.280155 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 115 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 115 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.280308  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 115 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.280442 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 114. Current term is 115. Ops: []
I1014 19:26:03.280472 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 115 candidate_status { last_received { term: 114 index: 72 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.280680 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 115: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:03.281092 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 115 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 115: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:03.281152 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 115 election: Election decided. Result: candidate lost.
I1014 19:26:03.281272  2776 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 114.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I1014 19:26:03.281453  1004 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:03.281538  2776 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Stepping down as leader of term 114
I1014 19:26:03.281605  2776 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:03.281621  1004 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Leader election lost for term 115. Reason: could not achieve majority
I1014 19:26:03.281777  2776 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 114.72, Last appended by leader: 72, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.281987  2776 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:03.281950 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 115 candidate_status { last_received { term: 114 index: 72 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.282034  2776 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:03.282117 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 115: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:03.282157  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 114 FOLLOWER]: Advancing to term 115
I1014 19:26:03.282471  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 115 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.282471 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 115 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 115: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:03.283102 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 115 candidate_status { last_received { term: 114 index: 72 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.284397 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:03.285758 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 115.
I1014 19:26:03.286192 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 115 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.286291 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won.
I1014 19:26:03.286548  2780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:03.286626  2780 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Leader election won for term 115
I1014 19:26:03.286777  2780 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:03.286988  2780 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 114.72, Last appended by leader: 72, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.287855  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.288100  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.379019 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 114 index: 72. Preceding OpId from leader: term: 115 index: 73. (index mismatch)
I1014 19:26:03.379644  2389 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I1014 19:26:03.381000  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.381341  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:03.381786 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.793s	user 0.002s	sys 0.000s
I1014 19:26:03.381906 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:03.382309  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.382654  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.383424 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 114 index: 72. Preceding OpId from leader: term: 115 index: 73. (index mismatch)
I1014 19:26:03.383715 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:03.384799  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.385007  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.386112  2780 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I1014 19:26:03.388509  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.388793  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.388615  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.388914  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.389845  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 115 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.390204  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.391054 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:03.393018 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:03.490756  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:03.490835  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:03.490908  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 115 FOLLOWER]: Advancing to term 116
I1014 19:26:03.492470  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.492872  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 116 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.493048  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 116 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.493537 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 116 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.493657 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 116 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.493749 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 116 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.493847 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 116 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.494159 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 116 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 116 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.494323 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 116 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 116 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.494395 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost.
I1014 19:26:03.494642  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:03.494707  2516 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I1014 19:26:03.514129 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 115. Current term is 116. Ops: []
I1014 19:26:03.514647  2780 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 115.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I1014 19:26:03.514914  2389 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 LEADER]: Stepping down as leader of term 115
I1014 19:26:03.514969  2389 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:03.515180  2389 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 115.74, Last appended by leader: 74, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.515343  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:03.515424  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:03.515592  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 115 FOLLOWER]: Advancing to term 116
I1014 19:26:03.573753  2776 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:03.573863  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:03.573935  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 115 FOLLOWER]: Advancing to term 116
I1014 19:26:03.575804  2776 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.576308  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 116 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.576505  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 116 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.578626 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 116 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.578809 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:03.578828 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 116 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.579033 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 116: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:03.579741 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 116 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 116: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:03.580955 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 116.
I1014 19:26:03.581262 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 116 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.581324 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 116 election: Election decided. Result: candidate won.
I1014 19:26:03.581557  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:03.581662  2776 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 FOLLOWER]: Leader election won for term 116
I1014 19:26:03.581787  2776 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:03.581961  2776 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 115.74, Last appended by leader: 74, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.583009  3024 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 116 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.583355  3024 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.616058  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:03.616147  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.169s (starting election)
I1014 19:26:03.616220  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 116 FOLLOWER]: Advancing to term 117
I1014 19:26:03.618230  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.618752  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 117 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.618947  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 117 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.619423 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 117 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.619657 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 117 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.619585 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 117 candidate_status { last_received { term: 115 index: 74 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.619758 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 116 FOLLOWER]: Advancing to term 117
I1014 19:26:03.619841 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:03.620019 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 117 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 117 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.622050 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 117.
I1014 19:26:03.622520 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 117 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.622584 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won.
I1014 19:26:03.622911  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.195s (election complete)
I1014 19:26:03.622979  2516 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 FOLLOWER]: Leader election won for term 117
I1014 19:26:03.623124  2516 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:03.623363  2516 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 115.74, Last appended by leader: 74, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.624634  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.624827  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.625062 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:03.627065 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:03.672307 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 116. Current term is 117. Ops: []
I1014 19:26:03.672809  3024 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I1014 19:26:03.672996  3024 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 117.75 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 74
I1014 19:26:03.673069  2776 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 LEADER]: Stepping down as leader of term 116
I1014 19:26:03.673141  2776 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:03.673337  2776 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 116.75, Last appended by leader: 75, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.673498  2776 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:03.673558  2776 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:03.673647  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 116 FOLLOWER]: Advancing to term 117
I1014 19:26:03.728204 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 117 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 116 index: 75. Preceding OpId from leader: term: 117 index: 75. (term mismatch)
I1014 19:26:03.728269 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 74
I1014 19:26:03.728312 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 116.75
I1014 19:26:03.728363 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 117 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:03.728914  1004 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I1014 19:26:03.730087  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.730350  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.731640  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.731902  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.732538  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.732723  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.732770 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 115 index: 74. Preceding OpId from leader: term: 117 index: 75. (index mismatch)
I1014 19:26:03.733381  3275 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I1014 19:26:03.734659  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.734879  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.735030  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.735271  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:03.818162  1004 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 117 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.818480  1004 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.832325  2776 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 117 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:03.832427  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:03.832523  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 117 FOLLOWER]: Advancing to term 118
I1014 19:26:03.834079  2776 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.834496  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 118 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.834686  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 118 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.835194 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 118 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.835383 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 118 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.835525 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 118 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.835717 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 118 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.835934 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 118 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 118 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.836180 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 118 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 118 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.836251 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost.
I1014 19:26:03.836575  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:03.836658  2776 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I1014 19:26:03.854970 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 117. Current term is 118. Ops: []
I1014 19:26:03.855516  2516 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 117.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I1014 19:26:03.855779  2516 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Stepping down as leader of term 117
I1014 19:26:03.855835  2516 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:03.856022  2516 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 117.75, Last appended by leader: 75, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.856210  2516 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:03.856259  2516 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:03.856395  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 117 FOLLOWER]: Advancing to term 118
I1014 19:26:03.920123  2389 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:03.920235  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:03.920347  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 117 FOLLOWER]: Advancing to term 118
I1014 19:26:03.922718  2389 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.923274  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 118 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.924176  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 118 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:03.924280 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 118 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.924474 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:03.924859 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 118 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:03.925065 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 118: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:03.925541 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 118 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 118: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:03.926653 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 118.
I1014 19:26:03.926964 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 118 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.927047 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 118 election: Election decided. Result: candidate won.
I1014 19:26:03.927318  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:03.927392  2389 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 FOLLOWER]: Leader election won for term 118
I1014 19:26:03.927534  2389 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:03.927747  2389 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 117.75, Last appended by leader: 75, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.928870  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 118 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.929090  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:03.940471  2776 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:03.940582  2776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.183s (starting election)
I1014 19:26:03.940651  2776 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 118 FOLLOWER]: Advancing to term 119
I1014 19:26:03.942903  2776 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.943506  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 119 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.944262 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 119 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:03.944352  2776 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 119 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:03.944463 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 118 FOLLOWER]: Advancing to term 119
I1014 19:26:03.944592 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:03.946310 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 119 candidate_status { last_received { term: 117 index: 75 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:03.946525 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 119 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.946717 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 119.
I1014 19:26:03.946976 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 119 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 119 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:03.947337 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 119 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:03.947420 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 119 election: Election decided. Result: candidate won.
I1014 19:26:03.947734  3024 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.214s (election complete)
I1014 19:26:03.947818  3024 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 FOLLOWER]: Leader election won for term 119
I1014 19:26:03.948348  3024 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:03.948652  3024 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 117.75, Last appended by leader: 75, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:03.950435  3399 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:03.950709  3399 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.042655 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 118. Current term is 119. Ops: []
I1014 19:26:04.043201  2780 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I1014 19:26:04.043503  2780 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 LEADER]: Stepping down as leader of term 118
I1014 19:26:04.043596  2780 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:04.043802  2780 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 118.76, Last appended by leader: 76, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.043953  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:04.044023  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:04.044143  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 118 FOLLOWER]: Advancing to term 119
I1014 19:26:04.048025 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:04.048990 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:04.050079 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 117 vs 119: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 119 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.050352 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 119 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.050668 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 119 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.053330 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 117 index: 75. Preceding OpId from leader: term: 119 index: 76. (index mismatch)
I1014 19:26:04.053834  3399 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I1014 19:26:04.055105  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.055333  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.056258  2776 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.056448  2776 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.056854  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.057039  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.057358 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 119 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 118 index: 76. Preceding OpId from leader: term: 119 index: 76. (term mismatch)
I1014 19:26:04.057420 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 75
I1014 19:26:04.057497 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 118.76
I1014 19:26:04.057556 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 119 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:04.058099  2776 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I1014 19:26:04.059392  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.059705  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.059826  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.060012  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.149333  3399 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 119 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.149641  3399 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.339821  2389 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 119 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:04.339926  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:04.340026  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 119 FOLLOWER]: Advancing to term 120
I1014 19:26:04.342281  2389 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.342861  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 120 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:04.343240  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 120 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.343556 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 120 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:04.343773 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 120 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.343859 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 120 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:04.344069 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 120 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.344250 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 120 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 120 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.344425 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 120 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 120 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.344517 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 120 election: Election decided. Result: candidate lost.
I1014 19:26:04.344846  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:04.344919  2389 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Leader election lost for term 120. Reason: could not achieve majority
I1014 19:26:04.352923 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 119. Current term is 120. Ops: []
I1014 19:26:04.353492  3024 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 119.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.353735  3399 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 LEADER]: Stepping down as leader of term 119
I1014 19:26:04.353801  3399 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:04.354032  3399 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 119.76, Last appended by leader: 76, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.354198  3399 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:04.354260  3399 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:04.354387  3399 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 119 FOLLOWER]: Advancing to term 120
I1014 19:26:04.405014  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:04.405120  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:04.405196  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 119 FOLLOWER]: Advancing to term 120
I1014 19:26:04.406805  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.407222  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 120 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.407400  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 120 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:04.408181 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 120 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:04.408181 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 120 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:04.408368 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (vote granted)
I1014 19:26:04.408396 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 120: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:04.408929 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 120 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 120: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:04.410089 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 120.
I1014 19:26:04.410419 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 120 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.410502 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 120 election: Election decided. Result: candidate won.
I1014 19:26:04.410980  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:04.411069  2516 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 FOLLOWER]: Leader election won for term 120
I1014 19:26:04.411510  2516 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:04.411733  2516 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 119.76, Last appended by leader: 76, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.412699  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 120 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.412956  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.413179 32444 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:04.414728 32444 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:04.493207  2389 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:04.493306  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:04.493391  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 120 FOLLOWER]: Advancing to term 121
I1014 19:26:04.495043  2389 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.495563  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 121 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:04.495769  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 121 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.496776 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 121 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:04.496999 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 121 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.497671 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 121 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 121 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.498005 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 121 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:04.498148 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 120 FOLLOWER]: Advancing to term 121
I1014 19:26:04.498255 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:04.499824 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 121.
I1014 19:26:04.500183 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 121 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.500273 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won.
I1014 19:26:04.500537  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:04.500612  2389 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 FOLLOWER]: Leader election won for term 121
I1014 19:26:04.500746  2389 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:04.500947  2389 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 119.76, Last appended by leader: 76, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.502485  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 121 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.502655  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:04.519199  2502 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.446s	user 0.001s	sys 0.001s
I1014 19:26:04.519265  2502 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:04.520015  2502 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:04.520846  2502 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 114 vs 121: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521085  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521253  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521463  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521631  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521822  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.521981  2502 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 121 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.531321 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 120. Current term is 121. Ops: []
I1014 19:26:04.531965  2516 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.532232  2516 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 121.77 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 76
I1014 19:26:04.532346  3558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 LEADER]: Stepping down as leader of term 120
I1014 19:26:04.532404  3558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:04.532532 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 121 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 120. Current term is 121. Ops: []
I1014 19:26:04.532632  3558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 120.77, Last appended by leader: 77, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.532855  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:04.532920  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:04.533025  3558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 120 FOLLOWER]: Advancing to term 121
I1014 19:26:04.624490 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 121 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 119 index: 76. Preceding OpId from leader: term: 121 index: 77. (index mismatch)
I1014 19:26:04.624505 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 120 index: 77. Preceding OpId from leader: term: 121 index: 77. (term mismatch)
I1014 19:26:04.624574 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 76
I1014 19:26:04.624610 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 120.77
I1014 19:26:04.624680 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:04.625255  2389 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.625645  2389 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.626102  3399 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 121 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:04.626232  3399 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:04.626307  3399 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 121 FOLLOWER]: Advancing to term 122
I1014 19:26:04.627080  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 121 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.627279  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.628476  3399 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.628535  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 121 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.628787  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.628885 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 121. Current term is 122. Ops: []
I1014 19:26:04.629055  3399 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 122 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:04.629248  3399 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 122 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:04.629771 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 122 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:04.630051 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 122 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:04.630108  2780 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 119.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.630362 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 122 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.630558  2780 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Stepping down as leader of term 121
I1014 19:26:04.630609  2780 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:04.631129 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 122 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 122 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.631275  2780 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 121.77, Last appended by leader: 77, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.631506  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:04.631572  2780 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:04.631671  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 121 FOLLOWER]: Advancing to term 122
I1014 19:26:04.638043 28639 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 122 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:04.638303  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 121 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.638556  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.638836 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 122 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 122 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:04.638937 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 122 election: Election decided. Result: candidate lost.
I1014 19:26:04.639175  3399 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:04.639284  3399 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Leader election lost for term 122. Reason: could not achieve majority
I1014 19:26:04.728734  3558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:04.728826  3558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:04.728904  3558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 121 FOLLOWER]: Advancing to term 122
I1014 19:26:04.730736  3558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.731245  3558 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 122 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.731467  3558 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 122 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:04.732424 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 122 candidate_status { last_received { term: 121 index: 77 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:04.732626 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:04.733669 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 122 candidate_status { last_received { term: 121 index: 77 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:04.733884 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 122: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:04.734326 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 122 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 122: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:04.734740 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 122.
I1014 19:26:04.735075 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 122 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:04.735152 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 122 election: Election decided. Result: candidate won.
I1014 19:26:04.735532  3558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:04.735627  3558 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 FOLLOWER]: Leader election won for term 122
I1014 19:26:04.735790  3558 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:04.736050  3558 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 121.77, Last appended by leader: 77, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.737108  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 122 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.737347  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.758865  3399 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:04.758967  3399 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:04.759033  3399 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 122 FOLLOWER]: Advancing to term 123
I1014 19:26:04.761070  3399 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.761610  3399 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 123 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:04.761847  3399 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 123 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:04.762485 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 123 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:04.762508 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 123 candidate_status { last_received { term: 119 index: 76 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:04.762712 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 123 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.762696 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 122 FOLLOWER]: Advancing to term 123
I1014 19:26:04.763159 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 123 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 123 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:04.764804 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 123 because replica has last-logged OpId of term: 121 index: 77, which is greater than that of the candidate, which has last-logged OpId of term: 119 index: 76.
I1014 19:26:04.765197 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 123 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 123 because replica has last-logged OpId of term: 121 index: 77, which is greater than that of the candidate, which has last-logged OpId of term: 119 index: 76.
I1014 19:26:04.765267 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost.
I1014 19:26:04.765601  3399 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.206s (election complete)
I1014 19:26:04.765692  3399 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
W1014 19:26:04.828984 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.436s	user 0.002s	sys 0.000s
I1014 19:26:04.829061 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:04.829964 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:04.831046 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 115 vs 123: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.831331 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.831583 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.831812 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.832026 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.832263 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.832540 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.832767 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.832999 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.833324 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.833595 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.833854 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.834100 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.834339 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.834658 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:04.834904 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 123 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.859592  2780 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:04.859719  2780 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:04.859836  2780 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 123 FOLLOWER]: Advancing to term 124
I1014 19:26:04.862963 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 123 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 122. Current term is 123. Ops: []
I1014 19:26:04.862884  2780 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.863343 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 122. Current term is 124. Ops: []
I1014 19:26:04.863674  3558 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.863932  3558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 LEADER]: Stepping down as leader of term 122
I1014 19:26:04.863994  3558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:04.864219  3558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 122.78, Last appended by leader: 78, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.864437  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:04.864511  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:04.864624  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 124 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:04.864648  3558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 122 FOLLOWER]: Advancing to term 123
I1014 19:26:04.864997  2780 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 124 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.865556 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 124 candidate_status { last_received { term: 121 index: 77 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:04.865713 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 123 FOLLOWER]: Advancing to term 124
I1014 19:26:04.865808 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.282s (vote granted)
I1014 19:26:04.867622 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 124 candidate_status { last_received { term: 121 index: 77 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:04.867777 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 123 FOLLOWER]: Advancing to term 124
I1014 19:26:04.867921 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 124.
I1014 19:26:04.868510 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 124 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:04.868595 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 124 election: Election decided. Result: candidate won.
I1014 19:26:04.868829  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:04.868909  2389 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 FOLLOWER]: Leader election won for term 124
I1014 19:26:04.869802 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 124 because replica has last-logged OpId of term: 122 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 121 index: 77.
I1014 19:26:04.870190  2389 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:04.870510  2389 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 121.77, Last appended by leader: 77, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:04.871781 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 124 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 124 because replica has last-logged OpId of term: 122 index: 78, which is greater than that of the candidate, which has last-logged OpId of term: 121 index: 77.
I1014 19:26:04.872254  2780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.872400  2780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.872661 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:04.874228 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:04.962599 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 122 index: 78. Preceding OpId from leader: term: 124 index: 78. (term mismatch)
I1014 19:26:04.962671 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 77
I1014 19:26:04.962713 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 122.78
I1014 19:26:04.962765 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:04.963354  2780 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I1014 19:26:04.964694  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.964915  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.966087  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.966320  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:04.967566 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 119 index: 76. Preceding OpId from leader: term: 124 index: 78. (index mismatch)
I1014 19:26:04.967948  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.968145  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.968345  2389 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 76, Time since last communication: 0.000s
I1014 19:26:04.971505  3828 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.971724  3828 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.971873  3399 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.972076  3399 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:04.972180  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:04.972457  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.066370  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 124 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.066757  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.344137  2516 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:05.344275  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:05.344357  2516 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 124 FOLLOWER]: Advancing to term 125
I1014 19:26:05.346658  2516 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.347215  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 125 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:05.347409  2516 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 125 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.348523 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 124. Current term is 125. Ops: []
I1014 19:26:05.349156  2389 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 124.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I1014 19:26:05.349442  2389 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 LEADER]: Stepping down as leader of term 124
I1014 19:26:05.349522  2389 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:05.349750  2389 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 124.78, Last appended by leader: 78, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.349970  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:05.350037  2389 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:05.350248  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 124 FOLLOWER]: Advancing to term 125
I1014 19:26:05.350301 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 125 candidate_status { last_received { term: 124 index: 78 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:05.350538 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 125 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.350477 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 125 candidate_status { last_received { term: 124 index: 78 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:05.350982 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 125 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 125 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.351594  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:05.351677  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:05.351763  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 124 FOLLOWER]: Advancing to term 125
I1014 19:26:05.352617 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:05.353806  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.354212  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 125 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.354372  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 125 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.354550 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 125.
I1014 19:26:05.354779 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 125 candidate_status { last_received { term: 124 index: 78 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:05.354912 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 125 candidate_status { last_received { term: 124 index: 78 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:05.355017 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 125: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:05.355017 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 125 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.355110 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 125: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:05.355129 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 125 election: Election decided. Result: candidate won.
I1014 19:26:05.355556  2516 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:05.355504 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 125 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 125: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:05.355643  2516 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Leader election won for term 125
I1014 19:26:05.355697 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 125 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 125: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:05.355741  2516 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:05.355800 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost.
I1014 19:26:05.356067  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:05.355937  2516 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 124.78, Last appended by leader: 78, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.356158  3829 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I1014 19:26:05.356916  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.357066  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.466708 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 124 index: 78. Preceding OpId from leader: term: 125 index: 79. (index mismatch)
I1014 19:26:05.467264  3558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I1014 19:26:05.468657  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.468948  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.470126  2516 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.470325 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 125 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 124 index: 78. Preceding OpId from leader: term: 125 index: 79. (index mismatch)
I1014 19:26:05.470430  2516 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.470557  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.470806  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.470909  3558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I1014 19:26:05.475383  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.475690  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.475888  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.476166  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.557792  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 125 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.558181  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.659543  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 125 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:05.659658  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:05.659744  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 125 FOLLOWER]: Advancing to term 126
I1014 19:26:05.662025  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.662592  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 126 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.662813  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 126 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.663502 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 126 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:05.663491 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 126 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:05.663703 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 126 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.663717 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 126 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.664186 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 126 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 126 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.664325 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 126 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 126 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.664412 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 126 election: Election decided. Result: candidate lost.
I1014 19:26:05.664656  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:05.664757  3976 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Leader election lost for term 126. Reason: could not achieve majority
I1014 19:26:05.672243 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 125. Current term is 126. Ops: []
I1014 19:26:05.673192  3558 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 125.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I1014 19:26:05.673481  3558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 LEADER]: Stepping down as leader of term 125
I1014 19:26:05.673542  3558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:05.673746  3558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 125.79, Last appended by leader: 79, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.673950  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:05.674016  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:05.674160  3558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 125 FOLLOWER]: Advancing to term 126
I1014 19:26:05.748705  2389 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:05.748816  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:26:05.748910  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 125 FOLLOWER]: Advancing to term 126
I1014 19:26:05.751304  2389 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.751880  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 126 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.752374  2389 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 126 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:05.752586 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 126 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:05.752780 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (vote granted)
I1014 19:26:05.754890 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 126.
I1014 19:26:05.755437 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 126 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:05.755676 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 126: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:05.756050 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 126 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 126: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:05.756206 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 126 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.756290 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 126 election: Election decided. Result: candidate won.
I1014 19:26:05.756700  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:05.756810  2389 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 FOLLOWER]: Leader election won for term 126
I1014 19:26:05.757050  2389 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:05.757381  2389 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 125.79, Last appended by leader: 79, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.759073  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 126 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.759249  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.772639  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:05.772763  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.194s (starting election)
I1014 19:26:05.772850  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 126 FOLLOWER]: Advancing to term 127
I1014 19:26:05.775035  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.775602  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 127 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.775938  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 127 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.776698 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 127 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:05.776700 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 127 candidate_status { last_received { term: 125 index: 79 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:05.776890 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 126 FOLLOWER]: Advancing to term 127
I1014 19:26:05.776933 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 127 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.776990 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (vote granted)
I1014 19:26:05.777403 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 127 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 127 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.779235 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 127.
I1014 19:26:05.779579 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 127 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:05.779666 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 127 election: Election decided. Result: candidate won.
I1014 19:26:05.780086  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.191s (election complete)
I1014 19:26:05.780162  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 FOLLOWER]: Leader election won for term 127
I1014 19:26:05.780315  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:05.780552  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 125.79, Last appended by leader: 79, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.781585  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.781775  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:05.781868 32444 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.367s	user 0.002s	sys 0.000s
I1014 19:26:05.781914 32444 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:05.782871 32444 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:05.784162 32444 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 120 vs 127: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 127 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:05.784610 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 127 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:05.784900 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 127 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:05.785362 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 127 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:05.785657 32444 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 127 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.786216  4127 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:05.790061  4127 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:05.846267 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 126. Current term is 127. Ops: []
I1014 19:26:05.846907  3757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I1014 19:26:05.847115  3757 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 127.80 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 79
I1014 19:26:05.847230  3757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 LEADER]: Stepping down as leader of term 126
I1014 19:26:05.847281  3757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:05.847499  3757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 126.80, Last appended by leader: 80, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.847709  3757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:05.847779  3757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:05.847906  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 126 FOLLOWER]: Advancing to term 127
I1014 19:26:05.867856 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 127 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 125 index: 79. Preceding OpId from leader: term: 127 index: 80. (index mismatch)
I1014 19:26:05.868371  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I1014 19:26:05.869576  4060 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.870069  4060 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.871167  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.871405  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.872217 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 126 index: 80. Preceding OpId from leader: term: 127 index: 80. (term mismatch)
I1014 19:26:05.872287 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 79
I1014 19:26:05.872335 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 126.80
I1014 19:26:05.872400 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:05.872995  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I1014 19:26:05.874821  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.875123  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.874930  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.875221  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.876330  4060 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.876556  4060 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:05.971093  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 127 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:05.971386  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:05.971854  4060 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 127 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:05.971954  4060 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:05.972041  4060 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 127 FOLLOWER]: Advancing to term 128
I1014 19:26:05.974309  4060 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.974831  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 128 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:05.975040  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 128 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:05.975489 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 128 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:05.975661 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 128 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.975577 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 128 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:05.975778 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 128 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.976047 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 128 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 128 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.976168 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 128 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 128 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:05.976253 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost.
I1014 19:26:05.976495  3558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:05.976609  3558 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I1014 19:26:05.984060 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 128 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 127. Current term is 128. Ops: []
I1014 19:26:05.984586  3976 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 127.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I1014 19:26:05.984799  3829 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Stepping down as leader of term 127
I1014 19:26:05.984859  3829 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:05.985086  3829 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 127.80, Last appended by leader: 80, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:05.985321  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:05.985365  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:05.985536  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 127 FOLLOWER]: Advancing to term 128
I1014 19:26:06.064115  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:06.064203  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:06.064270  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 128 FOLLOWER]: Advancing to term 129
I1014 19:26:06.066385  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.066929  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 129 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.067982 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 129 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.068136 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 128 FOLLOWER]: Advancing to term 129
I1014 19:26:06.068222 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (vote granted)
I1014 19:26:06.070216 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 129.
I1014 19:26:06.072999  3757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:06.073081  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:06.073155  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 127 FOLLOWER]: Advancing to term 128
I1014 19:26:06.075280  3757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.075850  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 128 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.076885  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 128 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.077344 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 129 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.077430 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 129 election: Election decided. Result: candidate won.
I1014 19:26:06.077827 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 128 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.077924  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:06.077996  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 FOLLOWER]: Leader election won for term 129
I1014 19:26:06.078022 28640 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 128. Current term is 129.
I1014 19:26:06.078102  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:06.078426  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 129 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
W1014 19:26:06.078469 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 128 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 128. Current term is 129.
I1014 19:26:06.078356  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 127.80, Last appended by leader: 80, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.078554 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 128 election: Cancelling election due to peer responding with higher term
I1014 19:26:06.078826  2389 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:06.078929  2389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 128 FOLLOWER]: Advancing to term 129
I1014 19:26:06.079262 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 129 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.079412  4272 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.079820  4272 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.081384  2389 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Leader election lost for term 128. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 128. Current term is 129.
I1014 19:26:06.081604 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.203s (vote granted)
I1014 19:26:06.083595 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 129.
I1014 19:26:06.084236 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 129 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.084973 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 128 candidate_status { last_received { term: 127 index: 80 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.085141 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 128 because replica is either leader or believes a valid leader to be alive.
W1014 19:26:06.085495 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 128 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 128 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.158234 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 127 index: 80. Preceding OpId from leader: term: 129 index: 81. (index mismatch)
I1014 19:26:06.158789  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I1014 19:26:06.159972  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.160178  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.161245  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.161460  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.161543 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 127 index: 80. Preceding OpId from leader: term: 129 index: 81. (index mismatch)
I1014 19:26:06.162569  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.162784  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.162964  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I1014 19:26:06.164307  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.164558  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.164690  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.164885  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:06.235172 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.361s	user 0.002s	sys 0.000s
I1014 19:26:06.235234 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:06.236177 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:06.238992 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 124 vs 129: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:06.239274 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:06.239643 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:06.239919 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.264863  4060 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:06.264946  4060 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:06.265040  4060 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 129 FOLLOWER]: Advancing to term 130
I1014 19:26:06.267191  4060 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.267763  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 130 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.267946  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 130 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.268451 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 130 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.268669 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 130 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.269090 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 130 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 130 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.269032 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 130 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.269242 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 130 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.269661 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 130 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 130 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.269765 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost.
I1014 19:26:06.270125  4060 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:06.270215  4060 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I1014 19:26:06.270639  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 129 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.270864  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.271008 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 129. Current term is 130. Ops: []
I1014 19:26:06.271447  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 129.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I1014 19:26:06.271734  3976 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Stepping down as leader of term 129
I1014 19:26:06.271821  3976 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:06.272028  3976 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 129.81, Last appended by leader: 81, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.272281  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:06.272333  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:06.272498  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 129 FOLLOWER]: Advancing to term 130
I1014 19:26:06.349838  3757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:06.349938  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:06.350029  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 129 FOLLOWER]: Advancing to term 130
I1014 19:26:06.352272  3757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.352826  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 130 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.352998  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 130 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.353817 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 130 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.354029 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:26:06.354355 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 130 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.354591 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 130: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:06.355147 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 130 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 130: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:06.356194 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 130.
I1014 19:26:06.356595 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 130 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.356667 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 130 election: Election decided. Result: candidate won.
I1014 19:26:06.357419  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:06.357576  3757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 FOLLOWER]: Leader election won for term 130
I1014 19:26:06.357820  3757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:06.358064  3757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 129.81, Last appended by leader: 81, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.359473  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 130 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.359704  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.359930 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:06.361764 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:06.378504  4060 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:06.378597  4060 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.174s (starting election)
I1014 19:26:06.378684  4060 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 130 FOLLOWER]: Advancing to term 131
I1014 19:26:06.380748  4060 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.381264  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 131 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.381531  4060 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 131 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.382041 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 131 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.382220 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 130 FOLLOWER]: Advancing to term 131
I1014 19:26:06.382160 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 131 candidate_status { last_received { term: 129 index: 81 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.382302 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (vote granted)
I1014 19:26:06.382403 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 131 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.382998 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 131 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 131 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.384382 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 131.
I1014 19:26:06.384845 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 131 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.384908 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won.
I1014 19:26:06.385134  4060 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:06.385202  4060 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 FOLLOWER]: Leader election won for term 131
I1014 19:26:06.385349  4060 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:06.385581  4060 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 129.81, Last appended by leader: 81, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.386838  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.387001  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.395190  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:06.397482  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:06.455970 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 131 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 130. Current term is 131. Ops: []
I1014 19:26:06.456567  3757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I1014 19:26:06.456823  3757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 LEADER]: Stepping down as leader of term 130
I1014 19:26:06.456885  3757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:06.457084  3757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 130.82, Last appended by leader: 82, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.457293  3757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:06.457370  3757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:06.457495  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 130 FOLLOWER]: Advancing to term 131
I1014 19:26:06.465023 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 131 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 129 index: 81. Preceding OpId from leader: term: 131 index: 82. (index mismatch)
I1014 19:26:06.465550  3558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I1014 19:26:06.467109  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.467322  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.468439  3558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.468679  3558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.468811  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.469009  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.469744 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 130 index: 82. Preceding OpId from leader: term: 131 index: 82. (term mismatch)
I1014 19:26:06.469804 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 81
I1014 19:26:06.469844 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 130.82
I1014 19:26:06.469900 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:06.470417  4060 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I1014 19:26:06.475029  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.475255  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.475381  2389 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.475605  2389 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.549891  4060 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 131 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.550200  4060 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.569362  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 131 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:06.569468  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:06.569556  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 131 FOLLOWER]: Advancing to term 132
I1014 19:26:06.572038  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.572415 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 131. Current term is 132. Ops: []
I1014 19:26:06.572827  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 132 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.573040  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 132 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.573701 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 132 candidate_status { last_received { term: 131 index: 82 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.573890 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 132 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.573864  3558 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 131.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I1014 19:26:06.574115  3558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 LEADER]: Stepping down as leader of term 131
I1014 19:26:06.574201  3558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:06.574235 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 132 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 132 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.574296 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 132 candidate_status { last_received { term: 131 index: 82 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.574460  3558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 131.82, Last appended by leader: 82, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.574723  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:06.574790  3558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:06.574925  3558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 131 FOLLOWER]: Advancing to term 132
I1014 19:26:06.576964 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (vote granted)
I1014 19:26:06.578294 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 132.
I1014 19:26:06.578902 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 132 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.578980 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 132 election: Election decided. Result: candidate won.
I1014 19:26:06.579200  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (election complete)
I1014 19:26:06.579327  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 FOLLOWER]: Leader election won for term 132
I1014 19:26:06.579417  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:06.579633  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 131.82, Last appended by leader: 82, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.580585  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.580768  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.651221  3757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:06.651331  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:06.651399  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 131 FOLLOWER]: Advancing to term 132
I1014 19:26:06.653645  3757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.654152  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 132 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.654338  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 132 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.655174 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 132 candidate_status { last_received { term: 131 index: 82 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.655369 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 132 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.655768 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 132 candidate_status { last_received { term: 131 index: 82 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.655908 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 132 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 132 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.656065 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 132: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:06.656417 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 132 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 132: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:06.656499 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 132 election: Election decided. Result: candidate lost.
I1014 19:26:06.656756  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:06.656831  3757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 132 FOLLOWER]: Leader election lost for term 132. Reason: could not achieve majority
I1014 19:26:06.703292 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 131 index: 82. Preceding OpId from leader: term: 132 index: 83. (index mismatch)
I1014 19:26:06.704104  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I1014 19:26:06.704758 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 132 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 131 index: 82. Preceding OpId from leader: term: 132 index: 83. (index mismatch)
I1014 19:26:06.705875  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.706120  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.706293  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I1014 19:26:06.707479  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.707700  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.707758  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.707943  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.709537  4272 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.709614  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.709772  4272 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:06.709800  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.710103  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 132 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.710388  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:06.810600  3757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 132 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:06.810714  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (starting election)
I1014 19:26:06.810788  3757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 132 FOLLOWER]: Advancing to term 133
I1014 19:26:06.811146  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:06.811223  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:06.811287  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 132 FOLLOWER]: Advancing to term 133
I1014 19:26:06.813035  3757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.813565  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 133 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.813446  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.813745  3757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 133 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.813977  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 133 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:06.814127  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 133 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.814713 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 133 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.814925 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 133 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.814893 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 133 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.814880 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 133 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.815078 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 133: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:06.815105 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 133: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:06.815496 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 133 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 133 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.815683 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 133 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 133: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:06.815698 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 133 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 133: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:06.815762 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost.
I1014 19:26:06.815867 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 133 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:06.816032 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 133 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.816263  3757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:06.816330 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 133 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 133 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:06.816382  3757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I1014 19:26:06.816403 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost.
I1014 19:26:06.816720  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:06.816792  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I1014 19:26:06.822350 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 132. Current term is 133. Ops: []
I1014 19:26:06.823194  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 132.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I1014 19:26:06.823441  3976 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Stepping down as leader of term 132
I1014 19:26:06.823518  3976 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:06.823738  3976 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 132.83, Last appended by leader: 83, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.823942  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:06.823998  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:06.824118  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 132 FOLLOWER]: Advancing to term 133
I1014 19:26:06.913933  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:06.914028  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:06.914124  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 133 FOLLOWER]: Advancing to term 134
I1014 19:26:06.916355  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.916891  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 134 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.917054  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 134 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.917661 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 134 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:06.917743 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 134 candidate_status { last_received { term: 132 index: 83 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:06.917865 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 133 FOLLOWER]: Advancing to term 134
I1014 19:26:06.917948 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.168s (vote granted)
I1014 19:26:06.917958 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 133 FOLLOWER]: Advancing to term 134
I1014 19:26:06.918041 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.194s (vote granted)
I1014 19:26:06.919636 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 134.
I1014 19:26:06.919966 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 134 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:06.920013 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 134.
I1014 19:26:06.920034 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 134 election: Election decided. Result: candidate won.
I1014 19:26:06.920415 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 134 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:06.920599  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:06.920689  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 FOLLOWER]: Leader election won for term 134
I1014 19:26:06.920857  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:06.921066  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 132.83, Last appended by leader: 83, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:06.922266  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:06.922493  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.015938 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 134 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 132 index: 83. Preceding OpId from leader: term: 134 index: 84. (index mismatch)
I1014 19:26:07.016458  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I1014 19:26:07.017767  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.018106  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.018812  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.019104  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.019412  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.019606 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 134 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 132 index: 83. Preceding OpId from leader: term: 134 index: 84. (index mismatch)
I1014 19:26:07.019625  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.020149  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I1014 19:26:07.022212  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.022439  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.022586  3757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.022760  3757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.119817  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 134 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.119916  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:07.120004  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 134 FOLLOWER]: Advancing to term 135
I1014 19:26:07.121583  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.122046  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 135 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.122149  4514 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 134 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.122241  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 135 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.122241  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:07.122315  4514 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 134 FOLLOWER]: Advancing to term 135
I1014 19:26:07.122845 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 135 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.123054 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 135 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.123447 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 135 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.123581 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 135 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 135 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.124136  4514 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.124481 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 135: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.124678  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 135 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.124858  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 135 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.124898 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 135 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 135: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.124979 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost.
I1014 19:26:07.125504 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 135 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.125530 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 135 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.125632  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:07.125665 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 135 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.125725  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I1014 19:26:07.125959 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 135: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.125983 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 135 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 135 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.126291 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 135 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 135: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.126353 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost.
I1014 19:26:07.126605  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:07.126678  4514 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I1014 19:26:07.133648  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 134 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.133863  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.134029 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 134. Current term is 135. Ops: []
I1014 19:26:07.134635  4272 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 134.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I1014 19:26:07.134711 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 134. Current term is 135. Ops: []
I1014 19:26:07.134848  4272 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Stepping down as leader of term 134
I1014 19:26:07.134917  4272 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:07.135164  4272 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 134.84, Last appended by leader: 84, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.135480  4272 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:07.135488  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 134.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I1014 19:26:07.135788  4272 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:07.135931  4272 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 134 FOLLOWER]: Advancing to term 135
W1014 19:26:07.167064  4127 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.377s	user 0.002s	sys 0.000s
I1014 19:26:07.167130  4127 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:07.168095  4127 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:07.169086  4127 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 127 vs 135: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.169312  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.169582  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.169781  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.169958  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.170159  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.170328  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.170529  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.170739  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.170935  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.171104  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:07.171305  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 135 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.217979  4272 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 135 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:07.218502  4272 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:07.218576  4272 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 135 FOLLOWER]: Advancing to term 136
I1014 19:26:07.220809  4272 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.221320  4272 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 136 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.221930  4272 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 136 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.222014 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 136 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.222185 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 135 FOLLOWER]: Advancing to term 136
I1014 19:26:07.222270 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.181s (vote granted)
I1014 19:26:07.222726 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 136 candidate_status { last_received { term: 134 index: 84 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.222898 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 135 FOLLOWER]: Advancing to term 136
I1014 19:26:07.222973 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.187s (vote granted)
I1014 19:26:07.224510 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 136.
I1014 19:26:07.224812 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 136 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.224879 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 136 election: Election decided. Result: candidate won.
I1014 19:26:07.225149 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 136.
I1014 19:26:07.225222  4272 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:07.225333  4272 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 FOLLOWER]: Leader election won for term 136
I1014 19:26:07.225615 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 136 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.225615  4272 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:07.225867  4272 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 134.84, Last appended by leader: 84, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.227113  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.227326  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.227561  4127 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:07.229418  4127 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:07.313699 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 136 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 134 index: 84. Preceding OpId from leader: term: 136 index: 85. (index mismatch)
I1014 19:26:07.314255  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I1014 19:26:07.315722  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.315955  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.316720  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.317021  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.317430 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 136 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 134 index: 84. Preceding OpId from leader: term: 136 index: 85. (index mismatch)
I1014 19:26:07.317469  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.317662  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.317967  4272 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I1014 19:26:07.319814  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.320116  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.319814  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 136 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.320253  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.417570  4514 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 136 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.417655  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:07.417729  4514 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 136 FOLLOWER]: Advancing to term 137
I1014 19:26:07.419991  4525 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 136 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.420068  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:07.419988  4514 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.420138  4525 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 136 FOLLOWER]: Advancing to term 137
I1014 19:26:07.420568  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 137 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.420779  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 137 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.421294 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 137 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.421335 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 137 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.421540 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 137 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.421963 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 137 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 137 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.422610  4525 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.422951 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 137: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.423110  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 137 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.423310 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 137 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 137: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.423344  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 137 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.423393 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost.
I1014 19:26:07.423678  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:07.423794  4514 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I1014 19:26:07.423790 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 137 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.423979 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 137 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.424068 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 137 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.424244 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 137: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.424262 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 137 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 137 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.424757 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 137 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 137: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.424850 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost.
I1014 19:26:07.425135  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:07.425209  4525 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I1014 19:26:07.430341 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 136. Current term is 137. Ops: []
I1014 19:26:07.430876  4272 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 136.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I1014 19:26:07.431151  3976 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Stepping down as leader of term 136
I1014 19:26:07.431207  3976 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:07.431418  3976 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 136.85, Last appended by leader: 85, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.431586  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:07.431640  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:07.431782  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 136 FOLLOWER]: Advancing to term 137
I1014 19:26:07.548992  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:07.549098  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:07.549176  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 137 FOLLOWER]: Advancing to term 138
I1014 19:26:07.550765  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.551162  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 138 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.551427  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 138 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.551636  4525 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:07.551754  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:07.551892  4525 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 137 FOLLOWER]: Advancing to term 138
I1014 19:26:07.551970 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 138 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.552127 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 138 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.552289 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 137 FOLLOWER]: Advancing to term 138
I1014 19:26:07.552376 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.177s (vote granted)
I1014 19:26:07.554131  4525 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.554518 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 138: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.554555 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 138.
I1014 19:26:07.554685  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 138 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.555033  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 138 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.555042 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 138 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.555241 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 138 election: Election decided. Result: candidate won.
I1014 19:26:07.555383 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 138 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.555737 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 138: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:07.555886 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 138 candidate_status { last_received { term: 136 index: 85 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.556066 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 138: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:07.556097 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 138 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 138: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:07.556340  4127 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:07.556339 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 138 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 138: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:07.556640 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 138 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 138: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:07.556726 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 138 election: Election decided. Result: candidate lost.
I1014 19:26:07.556782  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:07.556924  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 FOLLOWER]: Leader election won for term 138
I1014 19:26:07.557188  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:07.557601  4127 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:07.557446  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 136.85, Last appended by leader: 85, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.558105  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:07.558233  4525 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Leader election lost for term 138. Reason: could not achieve majority
I1014 19:26:07.558885  4272 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.559116  4272 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.560596 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 136 index: 85. Preceding OpId from leader: term: 138 index: 87. (index mismatch)
I1014 19:26:07.560619 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 136 index: 85. Preceding OpId from leader: term: 138 index: 87. (index mismatch)
I1014 19:26:07.561044  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I1014 19:26:07.561184  4272 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I1014 19:26:07.563238  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.563455  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.562996  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.563880  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.564524  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.564723  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.565125  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.565270  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.565563  4127 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:07.566804  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.567080  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.567495  4127 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:07.567651  3973 mvcc.cc:203] Tried to move safe_time back from 6306080225523187712 to 6306080225519616000. Current Snapshot: MvccSnapshot[committed={T|T < 6306080225523187712 or (T in {6306080225523187712})}]
I1014 19:26:07.568475  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 138 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.568766  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.666785  4514 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.666884  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:07.666975  4514 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 138 FOLLOWER]: Advancing to term 139
I1014 19:26:07.668606  4514 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.669104  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 139 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.669313  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 139 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.669929 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 139 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.669908 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 139 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.670121 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 139 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.670130 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 139 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.670279 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 138. Current term is 139. Ops: []
I1014 19:26:07.670614 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 139 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 139 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.670778 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 139 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 139 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.670850 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost.
I1014 19:26:07.670789  3976 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 138.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I1014 19:26:07.671002  3976 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 LEADER]: Stepping down as leader of term 138
I1014 19:26:07.671062  3976 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:07.671072  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:26:07.671134  4514 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I1014 19:26:07.671278  3976 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 138.87, Last appended by leader: 87, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.671433  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:07.671492  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:07.671633  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 138 FOLLOWER]: Advancing to term 139
I1014 19:26:07.741724  4525 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:07.741837  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:07.741924  4525 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 138 FOLLOWER]: Advancing to term 139
I1014 19:26:07.744189  4525 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.744729  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 139 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.744906  4525 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 139 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.745489 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 139 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.745586 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 139 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.745690 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:07.745793 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 139: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.746192 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 139 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 139: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:07.747328 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 139.
I1014 19:26:07.747628 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 139 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.747701 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won.
I1014 19:26:07.747933  4525 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:07.748036  4525 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 FOLLOWER]: Leader election won for term 139
I1014 19:26:07.748181  4525 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:07.748386  4525 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 138.87, Last appended by leader: 87, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.749420  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 139 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.749641  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.798413  4514 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:07.798532  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:07.798614  4514 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 139 FOLLOWER]: Advancing to term 140
I1014 19:26:07.800132  4514 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.800566  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 140 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:07.800729  4514 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 140 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.801300 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 140 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:07.801367 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 140 candidate_status { last_received { term: 138 index: 87 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.801502 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 140 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.801559 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 139 FOLLOWER]: Advancing to term 140
I1014 19:26:07.801631 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:07.801856 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 140 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 140 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.803133 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 140.
I1014 19:26:07.803458 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 140 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.803529 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 140 election: Election decided. Result: candidate won.
I1014 19:26:07.803758  4514 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:26:07.803838  4514 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 FOLLOWER]: Leader election won for term 140
I1014 19:26:07.804299  4514 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:07.804656  4514 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 138.87, Last appended by leader: 87, Current term: 140, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.805465  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.806463  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.856705 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 139. Current term is 140. Ops: []
I1014 19:26:07.857331  4410 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I1014 19:26:07.857607  4525 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 LEADER]: Stepping down as leader of term 139
I1014 19:26:07.857664  4525 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:07.857875  4525 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 139.88, Last appended by leader: 88, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.858057  4525 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:07.858114  4525 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:07.858228  4525 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 139 FOLLOWER]: Advancing to term 140
I1014 19:26:07.885836 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 138 index: 87. Preceding OpId from leader: term: 140 index: 88. (index mismatch)
I1014 19:26:07.886382  4550 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I1014 19:26:07.887800  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.888016  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.889149  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.889153  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.889385  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.889433  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:07.890079 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 140 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 139 index: 88. Preceding OpId from leader: term: 140 index: 88. (term mismatch)
I1014 19:26:07.890149 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 87
I1014 19:26:07.890182 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 139.88
I1014 19:26:07.890230 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 140 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:07.890738  4514 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I1014 19:26:07.892102  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.892102  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:07.892463  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.892462  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:07.992283  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 140 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:07.992380  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:07.992470  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 140 FOLLOWER]: Advancing to term 141
I1014 19:26:07.994011  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:07.994395  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 141 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:07.994607  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 141 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:07.995061 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 141 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:07.995278 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 141 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.995256 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 141 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:07.995451 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 141 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.995914 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 141 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 141 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.996011 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 141 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 141 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:07.996103 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 141 election: Election decided. Result: candidate lost.
I1014 19:26:07.996431  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:07.996515  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Leader election lost for term 141. Reason: could not achieve majority
I1014 19:26:08.012835  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.013208  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.013242 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 140. Current term is 141. Ops: []
I1014 19:26:08.013715  4555 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 140.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I1014 19:26:08.013939  4555 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 LEADER]: Stepping down as leader of term 140
I1014 19:26:08.014010  4555 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:08.014202  4555 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 140.88, Last appended by leader: 88, Current term: 140, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.014374  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:08.014432  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:08.014569  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 140 FOLLOWER]: Advancing to term 141
I1014 19:26:08.051409  4127 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:08.052315  4127 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:08.053393  4127 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 138 vs 140: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.053686  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.053933  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.054152  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 140 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.082065  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:08.082167  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:08.082242  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 140 FOLLOWER]: Advancing to term 141
I1014 19:26:08.083860  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.084271  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 141 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.084501  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 141 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.085058 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 141 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.085235 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 141: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:08.085145 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 141 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.085331 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:08.085635 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 141 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 141: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:08.087702 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 141.
I1014 19:26:08.088024 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 141 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.088094 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 141 election: Election decided. Result: candidate won.
I1014 19:26:08.088335  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:08.088420  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 FOLLOWER]: Leader election won for term 141
I1014 19:26:08.088536  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.088762  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 140.88, Last appended by leader: 88, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.089604  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 141 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.089790  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.089993  4127 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:08.091480  4127 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:08.104315 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.742s	user 0.001s	sys 0.000s
I1014 19:26:08.104372 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:08.105130 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:08.106029 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 130 vs 141: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 141 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.106330 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 141 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.106552 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 141 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:08.106794 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 141 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.108067  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:08.108171  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.160s (starting election)
I1014 19:26:08.108242  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 141 FOLLOWER]: Advancing to term 142
I1014 19:26:08.110404  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.110904  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 142 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.111114  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 142 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.111623 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 142 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.111791 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 142 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.111735 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 142 candidate_status { last_received { term: 140 index: 88 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.111876 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 141 FOLLOWER]: Advancing to term 142
I1014 19:26:08.111961 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:08.112116 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 142 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 142 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.113972 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 142.
I1014 19:26:08.114244 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 142 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.114312 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 142 election: Election decided. Result: candidate won.
I1014 19:26:08.114522  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:08.114604  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 FOLLOWER]: Leader election won for term 142
I1014 19:26:08.114702  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:08.114943  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 140.88, Last appended by leader: 88, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.115967  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.116189  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:08.155128  4371 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.758s	user 0.000s	sys 0.001s
I1014 19:26:08.155194  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:08.155925  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:08.158038 28826 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 LEADER]: Stepping down as leader of term 141
I1014 19:26:08.158112 28826 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.158164 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 142 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 140 index: 88. Preceding OpId from leader: term: 142 index: 90. (index mismatch)
I1014 19:26:08.158331 28826 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 141.89, Last appended by leader: 89, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.158609 28826 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:08.158666 28826 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:08.158751 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 141 FOLLOWER]: Advancing to term 142
I1014 19:26:08.158721  4525 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I1014 19:26:08.160322  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.160691  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.160830 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 141 index: 89. Preceding OpId from leader: term: 142 index: 90. (index mismatch)
I1014 19:26:08.161520  4410 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I1014 19:26:08.161808  4410 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 141.89 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 88
I1014 19:26:08.162569 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 88
I1014 19:26:08.162622 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 141.89
I1014 19:26:08.162685 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:08.163203  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:08.163203  4525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.163457  4525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.163766  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.164127  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.164463  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.164743  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.166507  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:08.167675  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.167934  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.241197  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 142 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.241982  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.262683  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 142 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:08.262806  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:08.262882  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 142 FOLLOWER]: Advancing to term 143
I1014 19:26:08.264570  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.264981  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 143 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.265144  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 143 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.265681 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 143 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.265808 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 143 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.265899 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 143 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.265996 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 143 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.266388 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 143 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 143 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.266579 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 143 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 143 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.266654 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost.
I1014 19:26:08.266863  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:08.266974  4555 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I1014 19:26:08.275532 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 142. Current term is 143. Ops: []
I1014 19:26:08.276041  4525 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 142.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I1014 19:26:08.276368  4410 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 LEADER]: Stepping down as leader of term 142
I1014 19:26:08.276439  4410 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:08.276672  4410 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 142.90, Last appended by leader: 90, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.276834  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:08.276890  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:08.277021  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 142 FOLLOWER]: Advancing to term 143
I1014 19:26:08.342401  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:08.342509  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:08.342586  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 142 FOLLOWER]: Advancing to term 143
I1014 19:26:08.344032  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.344432  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 143 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.344655  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 143 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.345366 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 143 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.345366 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 143 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.345564 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:08.345635 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 143: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:08.347079 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 143 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 143: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:08.347158 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 143.
I1014 19:26:08.347476 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 143 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.347559 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 143 election: Election decided. Result: candidate won.
I1014 19:26:08.347779  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:08.347858  3829 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 FOLLOWER]: Leader election won for term 143
I1014 19:26:08.348001  3829 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.348193  3829 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 142.90, Last appended by leader: 90, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.349323  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 143 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.349490  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.390131  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:08.390249  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:08.390338  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 143 FOLLOWER]: Advancing to term 144
I1014 19:26:08.391942  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.392356  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 144 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.392575  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 144 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.393106 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 144 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.393296 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 143 FOLLOWER]: Advancing to term 144
I1014 19:26:08.393383 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (vote granted)
I1014 19:26:08.393503 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 144 candidate_status { last_received { term: 142 index: 90 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.393676 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 144 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.394035 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 144 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 144 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.395505 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 144.
I1014 19:26:08.395810 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 144 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.395880 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 144 election: Election decided. Result: candidate won.
I1014 19:26:08.396147  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.212s (election complete)
I1014 19:26:08.396231  4555 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 FOLLOWER]: Leader election won for term 144
I1014 19:26:08.396414  4555 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:08.396621  4555 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 142.90, Last appended by leader: 90, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.397548  4514 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.397771  4514 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.397980 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:08.400346 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:08.434916 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 143. Current term is 144. Ops: []
I1014 19:26:08.435500  3976 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I1014 19:26:08.435699  3976 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 144.91 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 90
I1014 19:26:08.435781  3829 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 LEADER]: Stepping down as leader of term 143
I1014 19:26:08.435842  3829 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.436054  3829 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 143.91, Last appended by leader: 91, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.436210  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:08.436272  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:08.436363  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 143 FOLLOWER]: Advancing to term 144
I1014 19:26:08.476238 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 144 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 142 index: 90. Preceding OpId from leader: term: 144 index: 91. (index mismatch)
I1014 19:26:08.476866  4514 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I1014 19:26:08.478262  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.478658  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.479998  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.480214  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.480557 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 143 index: 91. Preceding OpId from leader: term: 144 index: 91. (term mismatch)
I1014 19:26:08.480631 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 90
I1014 19:26:08.480701 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 143.91
I1014 19:26:08.480793 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:08.481065  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.481257  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.481564  4550 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I1014 19:26:08.483028  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.483291  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.483211  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.483508  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.559054  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 144 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.559348  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.581189  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 144 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:08.581313  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:08.581436  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 144 FOLLOWER]: Advancing to term 145
I1014 19:26:08.583765  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.584331  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 145 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.584615  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 145 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.585130 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 145 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.585320 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 145 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.585363 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 145 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.585600 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 145 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.585717 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 145 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 145 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.585960 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 145 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 145 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.586037 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost.
I1014 19:26:08.586299  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:08.586478  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I1014 19:26:08.599361 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 144. Current term is 145. Ops: []
I1014 19:26:08.599978  4550 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 144.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I1014 19:26:08.600255  4555 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Stepping down as leader of term 144
I1014 19:26:08.600352  4555 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:08.600625  4555 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 144.91, Last appended by leader: 91, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.600862  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:08.600971  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:08.601171  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 144 FOLLOWER]: Advancing to term 145
I1014 19:26:08.660840  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:08.660933  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:08.661012  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 144 FOLLOWER]: Advancing to term 145
I1014 19:26:08.663018  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.663627  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 145 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.663800  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 145 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.664496 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 145 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.664700 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 145: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:08.664701 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 145 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.664899 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (vote granted)
I1014 19:26:08.665052 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 145 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 145: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:08.666335 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 145.
I1014 19:26:08.666667 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 145 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.666733 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 145 election: Election decided. Result: candidate won.
I1014 19:26:08.666946  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:08.667021  3829 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 FOLLOWER]: Leader election won for term 145
I1014 19:26:08.667162  3829 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.667379  3829 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 144.91, Last appended by leader: 91, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.668226  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 145 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.668519  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.716645  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:08.716744  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:08.716816  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 145 FOLLOWER]: Advancing to term 146
I1014 19:26:08.718614  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.719149  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 146 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.719431  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 146 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.720093 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 146 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.720301 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 146 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.720304 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 146 candidate_status { last_received { term: 144 index: 91 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:08.720459 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 145 FOLLOWER]: Advancing to term 146
I1014 19:26:08.720566 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (vote granted)
I1014 19:26:08.721613 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 146 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 146 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.722365 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 146.
I1014 19:26:08.722784 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 146 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:08.722856 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 146 election: Election decided. Result: candidate won.
I1014 19:26:08.723249  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.218s (election complete)
I1014 19:26:08.723320  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 FOLLOWER]: Leader election won for term 146
I1014 19:26:08.723443  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:08.723619  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 144.91, Last appended by leader: 91, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.724498  4579 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.724865  4579 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.764284 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 145. Current term is 146. Ops: []
I1014 19:26:08.764847  3976 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I1014 19:26:08.765110  3976 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 146.92 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 91
I1014 19:26:08.765252  3829 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 LEADER]: Stepping down as leader of term 145
I1014 19:26:08.765308  3829 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:08.765560  3829 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 145.92, Last appended by leader: 92, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.765709  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:08.765774  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:08.765885  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 145 FOLLOWER]: Advancing to term 146
I1014 19:26:08.811388 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 145 index: 92. Preceding OpId from leader: term: 146 index: 92. (term mismatch)
I1014 19:26:08.811503 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 91
I1014 19:26:08.811547 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 145.92
I1014 19:26:08.811619 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:08.813127  4579 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I1014 19:26:08.814576  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.814798  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.815755  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.815914  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.816185 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 146 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 144 index: 91. Preceding OpId from leader: term: 146 index: 92. (index mismatch)
I1014 19:26:08.816344  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.816620  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.817035  4579 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I1014 19:26:08.818400  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.818734  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.818507  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.818826  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:08.918416  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 146 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:08.918545  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:08.918632  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 146 FOLLOWER]: Advancing to term 147
I1014 19:26:08.920172  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.920621  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 147 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:08.920838  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 147 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:08.921473 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 147 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:08.921520 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 147 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:08.921655 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 147 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.921735 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 147 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.922008 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 147 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 147 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.922196 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 147 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 147 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:08.922269 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 147 election: Election decided. Result: candidate lost.
I1014 19:26:08.922507  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:08.922616  4555 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Leader election lost for term 147. Reason: could not achieve majority
I1014 19:26:08.924350 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 146. Current term is 147. Ops: []
I1014 19:26:08.923933  4579 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 146 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:08.924679  4579 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:08.924918  4410 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 146.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I1014 19:26:08.925221  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Stepping down as leader of term 146
I1014 19:26:08.925282  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:08.925561  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 146.92, Last appended by leader: 92, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:08.925776  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:08.925840  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:08.925968  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 146 FOLLOWER]: Advancing to term 147
I1014 19:26:09.006033  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:09.006127  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:09.006216  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 146 FOLLOWER]: Advancing to term 147
I1014 19:26:09.007719  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.008110  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 147 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.008257  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 147 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.008987 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 147 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.008987 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 147 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.009193 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 147: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:09.009171 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:09.010756 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 147.
I1014 19:26:09.011153 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 147 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.011229 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 147 election: Election decided. Result: candidate won.
I1014 19:26:09.011515 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 147 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 147: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:09.011752  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:09.011819  3829 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 FOLLOWER]: Leader election won for term 147
I1014 19:26:09.011958  3829 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:09.012161  3829 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 146.92, Last appended by leader: 92, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.013239  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 147 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.013495  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.042554  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:09.042683  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.165s (starting election)
I1014 19:26:09.042762  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 147 FOLLOWER]: Advancing to term 148
I1014 19:26:09.044276  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.044720  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 148 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.044940  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 148 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.045667 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 148 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.045851 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 147 FOLLOWER]: Advancing to term 148
I1014 19:26:09.045933 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:09.047335 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 148.
I1014 19:26:09.047673 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 148 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.047742 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won.
I1014 19:26:09.047960  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.156s (election complete)
I1014 19:26:09.048040  4555 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 FOLLOWER]: Leader election won for term 148
I1014 19:26:09.048197  4555 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:09.048401  4555 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 146.92, Last appended by leader: 92, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.048650 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 148 candidate_status { last_received { term: 146 index: 92 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.048847 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 148 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.049278 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 148 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 148 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.049656  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.049829  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.115272 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 147. Current term is 148. Ops: []
I1014 19:26:09.115952  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I1014 19:26:09.116241  3976 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 LEADER]: Stepping down as leader of term 147
I1014 19:26:09.116309  3976 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:09.116540  3976 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 147.93, Last appended by leader: 93, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.116710  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:09.116775  3976 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:09.116897  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 147 FOLLOWER]: Advancing to term 148
I1014 19:26:09.163151  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:09.163978  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:09.164976  4371 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 142 vs 148: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.165030 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 148 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 147 index: 93. Preceding OpId from leader: term: 148 index: 93. (term mismatch)
I1014 19:26:09.165098 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 92
I1014 19:26:09.165136 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 147.93
I1014 19:26:09.165201 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 148 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
W1014 19:26:09.165374  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.165724  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.165917  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.165796  4550 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
W1014 19:26:09.166100  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.166286  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.166522  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.166800  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.167109  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.167423  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:09.167517  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 148 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.168514  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.168766  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.168742  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.168972 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 146 index: 92. Preceding OpId from leader: term: 148 index: 93. (index mismatch)
I1014 19:26:09.169003  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.169687  4555 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I1014 19:26:09.171023  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.171352  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.171561  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.171694  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.257549  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 148 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.257831  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.269260  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 148 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:09.269379  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:09.269448  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 148 FOLLOWER]: Advancing to term 149
I1014 19:26:09.271499  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.271925  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 149 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.272143  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 149 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.272717 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 149 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.272773 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 149 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.272884 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 149 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.272959 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 149 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.273198 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 149 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 149 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.273435 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 149 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 149 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.273525 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 149 election: Election decided. Result: candidate lost.
I1014 19:26:09.273739  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:09.273864  3976 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Leader election lost for term 149. Reason: could not achieve majority
I1014 19:26:09.290001 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 148. Current term is 149. Ops: []
I1014 19:26:09.290573  4550 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 148.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I1014 19:26:09.290834  4555 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 LEADER]: Stepping down as leader of term 148
I1014 19:26:09.290895  4555 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:09.291138  4555 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 148.93, Last appended by leader: 93, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.291345  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:09.291445  4555 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:09.291604  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 148 FOLLOWER]: Advancing to term 149
I1014 19:26:09.359238  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:09.359333  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:09.359416  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 148 FOLLOWER]: Advancing to term 149
I1014 19:26:09.360963  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.361413  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 149 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.361719  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 149 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.362254 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 149 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.362501 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 149: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:09.362756 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 149 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.362900 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 149 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 149: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:09.362929 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (vote granted)
I1014 19:26:09.364948 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 149.
I1014 19:26:09.365236 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 149 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.365305 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 149 election: Election decided. Result: candidate won.
I1014 19:26:09.365571  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:09.365645  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 FOLLOWER]: Leader election won for term 149
I1014 19:26:09.365790  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:09.365999  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 148.93, Last appended by leader: 93, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.367049  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 149 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.367254  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.367524  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:09.369618  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:09.391921  3976 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:09.392014  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:09.392087  3976 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 149 FOLLOWER]: Advancing to term 150
I1014 19:26:09.393714  3976 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.394209  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 150 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.394431  3976 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 150 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.395007 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 150 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.395237 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 150 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.395169 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 150 candidate_status { last_received { term: 148 index: 93 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.395336 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 149 FOLLOWER]: Advancing to term 150
I1014 19:26:09.395431 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:09.396005 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 150 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 150 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.397606 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 150.
I1014 19:26:09.398000 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 150 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.398069 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 150 election: Election decided. Result: candidate won.
I1014 19:26:09.398317  3976 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:09.398397  3976 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 FOLLOWER]: Leader election won for term 150
I1014 19:26:09.398524  3976 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:09.398737  3976 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 148.93, Last appended by leader: 93, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.399597  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.399822  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.475988 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 149. Current term is 150. Ops: []
I1014 19:26:09.476611  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I1014 19:26:09.476835  4647 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 150.94 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 93
I1014 19:26:09.476951  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 LEADER]: Stepping down as leader of term 149
I1014 19:26:09.477010  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:09.477222  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 149.94, Last appended by leader: 94, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.477412  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:09.477509  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:09.477639  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 149 FOLLOWER]: Advancing to term 150
I1014 19:26:09.484339 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 150 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 148 index: 93. Preceding OpId from leader: term: 150 index: 94. (index mismatch)
I1014 19:26:09.485386  3829 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I1014 19:26:09.486968  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.487227  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.488025  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.488250  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.488415  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.488620  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.489034 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 149 index: 94. Preceding OpId from leader: term: 150 index: 94. (term mismatch)
I1014 19:26:09.489104 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 93
I1014 19:26:09.489137 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 149.94
I1014 19:26:09.489192 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:09.489742  3976 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I1014 19:26:09.491142  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.491508  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.491803  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.491994  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:09.540431 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.140s	user 0.001s	sys 0.000s
I1014 19:26:09.540536 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:09.541700 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:09.542870 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 144 vs 150: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.543182 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.543664 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.543939 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.544229 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:09.544559 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.576148  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 150 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.576419  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.588954  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 150 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:09.589058  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:09.589148  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 150 FOLLOWER]: Advancing to term 151
I1014 19:26:09.591639  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.592202  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 151 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.592464  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 151 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.592943 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 151 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.593055 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 151 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.593173 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 151 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.593274 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 151 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.593555 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 151 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 151 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.593878 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 151 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 151 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.593951 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost.
I1014 19:26:09.594228  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:09.594305  4555 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I1014 19:26:09.613610 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 150. Current term is 151. Ops: []
I1014 19:26:09.614203  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 150.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I1014 19:26:09.614496  3829 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Stepping down as leader of term 150
I1014 19:26:09.614583  3829 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:09.614814  3829 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 150.94, Last appended by leader: 94, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.615087  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:09.615147  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:09.615316  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 150 FOLLOWER]: Advancing to term 151
I1014 19:26:09.678516  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:09.678647  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:09.678732  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 150 FOLLOWER]: Advancing to term 151
I1014 19:26:09.681067  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.681701  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 151 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.682112  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 151 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.682504 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 151 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.682689 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (vote granted)
I1014 19:26:09.682823 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 151 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.683035 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 151: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:09.683522 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 151 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 151: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:09.684842 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 151.
I1014 19:26:09.685138 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 151 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.685237 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won.
I1014 19:26:09.685614  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:09.685700  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 FOLLOWER]: Leader election won for term 151
I1014 19:26:09.685817  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:09.686039  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 150.94, Last appended by leader: 94, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.687301  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 151 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.687556  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.697230  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:09.697322  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.203s (starting election)
I1014 19:26:09.697402  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 151 FOLLOWER]: Advancing to term 152
I1014 19:26:09.699780  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.700351  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 152 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.700628  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 152 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.701169 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 152 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.701215 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 152 candidate_status { last_received { term: 150 index: 94 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.701380 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 152 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.701382 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 151 FOLLOWER]: Advancing to term 152
I1014 19:26:09.701493 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:09.703205 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 152.
I1014 19:26:09.703624 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 152 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.703711 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 152 election: Election decided. Result: candidate won.
I1014 19:26:09.703954  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.173s (election complete)
I1014 19:26:09.704038  4555 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 FOLLOWER]: Leader election won for term 152
I1014 19:26:09.704155  4555 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:09.704380  4555 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 150.94, Last appended by leader: 94, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.705513  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.705763 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 152 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 152 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.706056  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.706344 32481 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:09.708407 32481 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:09.770215 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 151. Current term is 152. Ops: []
I1014 19:26:09.770884  4410 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I1014 19:26:09.771123  4410 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 152.95 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 94
I1014 19:26:09.771229  4410 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 LEADER]: Stepping down as leader of term 151
I1014 19:26:09.771277  4410 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:09.771468  4410 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 151.95, Last appended by leader: 95, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.771672  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:09.771735  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:09.771857  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 151 FOLLOWER]: Advancing to term 152
I1014 19:26:09.791651 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 150 index: 94. Preceding OpId from leader: term: 152 index: 95. (index mismatch)
I1014 19:26:09.792135  4555 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I1014 19:26:09.793387  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.793727  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.794945  4550 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.795176  4550 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.795315 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 152 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 151 index: 95. Preceding OpId from leader: term: 152 index: 95. (term mismatch)
I1014 19:26:09.795384 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 94
I1014 19:26:09.795444 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 151.95
I1014 19:26:09.795507 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 152 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:09.795254  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.795612  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.796100  4550 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I1014 19:26:09.797610  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.797838  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.798054  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.798336  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:09.898008  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 152 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:09.898110  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:09.898185  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 152 FOLLOWER]: Advancing to term 153
I1014 19:26:09.900431  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.901017  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 153 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:09.901242  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 153 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.901798 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 153 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:09.902002 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 153 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.901949 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 153 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.902153 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 153 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.902499 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 153 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 153 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.902683 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 153 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 153 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:09.902752 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost.
I1014 19:26:09.902984  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:09.903057  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I1014 19:26:09.915979  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 152 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.916301  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:09.916712 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 152. Current term is 153. Ops: []
I1014 19:26:09.917546  4550 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 152.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I1014 19:26:09.917810  4550 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Stepping down as leader of term 152
I1014 19:26:09.917882  4550 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:09.918107  4550 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 152.95, Last appended by leader: 95, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.918337  4550 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:09.918404  4550 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:09.918606  4550 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 152 FOLLOWER]: Advancing to term 153
I1014 19:26:09.981240  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:09.981356  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:09.981429  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 152 FOLLOWER]: Advancing to term 153
I1014 19:26:09.983711  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.984232  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 153 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:09.984865  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 153 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.985057 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 153 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:09.985301 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 153: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:09.985550 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 153 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:09.985767 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
I1014 19:26:09.985908 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 153 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 153: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:09.988067 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 153.
I1014 19:26:09.988504 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 153 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:09.988610 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 153 election: Election decided. Result: candidate won.
I1014 19:26:09.988863  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:09.988946  3829 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 FOLLOWER]: Leader election won for term 153
I1014 19:26:09.989045  3829 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:09.989248  3829 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 152.95, Last appended by leader: 95, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:09.990226  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 153 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:09.990584  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.005844  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:10.005964  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:10.006042  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 153 FOLLOWER]: Advancing to term 154
I1014 19:26:10.008313  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.008906  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 154 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.009143  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 154 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.009737 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 154 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.009944 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 154 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.009860 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 154 candidate_status { last_received { term: 152 index: 95 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.010035 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 153 FOLLOWER]: Advancing to term 154
I1014 19:26:10.010124 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:10.012344 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 154.
I1014 19:26:10.013069 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 154 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.013181 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 154 election: Election decided. Result: candidate won.
I1014 19:26:10.013564 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 154 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 154 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.013564  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:10.013761  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 FOLLOWER]: Leader election won for term 154
I1014 19:26:10.014026  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.014300  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 152.95, Last appended by leader: 95, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.015774  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.015985  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:10.017441  4127 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.926s	user 0.001s	sys 0.000s
I1014 19:26:10.017530  4127 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:10.018563  4127 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:10.021219 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 154 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 153. Current term is 154. Ops: []
I1014 19:26:10.021714 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 153. Current term is 154. Ops: []
I1014 19:26:10.021803  3829 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I1014 19:26:10.022032  3829 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 LEADER]: Stepping down as leader of term 153
I1014 19:26:10.022131  3829 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:10.022334  3829 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 153.97, Last appended by leader: 97, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.022573  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:10.022645  3829 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:10.022794  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 153 FOLLOWER]: Advancing to term 154
I1014 19:26:10.123337 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 154 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 152 index: 95. Preceding OpId from leader: term: 154 index: 96. (index mismatch)
I1014 19:26:10.123945  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I1014 19:26:10.125264  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.125540  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.126529  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.126760  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.127118  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.127372  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.127615 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 154 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 153 index: 97. Preceding OpId from leader: term: 154 index: 96. (term mismatch)
I1014 19:26:10.127691 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 95
I1014 19:26:10.127737 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 153.96
I1014 19:26:10.127802 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 154 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:10.127876 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 153.97
I1014 19:26:10.128492  4127 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 141 vs 154: Aborted: Transaction aborted by new leader
W1014 19:26:10.128813  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.129057  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.129294  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.129570  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.129793  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.130024  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.130264  4127 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 154 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.130599  4410 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I1014 19:26:10.132194  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.132410  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.132577  3976 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.132751  3976 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.227809  4555 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 154 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.227919  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:10.228001  4555 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 154 FOLLOWER]: Advancing to term 155
I1014 19:26:10.229964  4555 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.230530  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 155 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.230751  4555 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 155 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.231278 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 155 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.231511 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 155 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.231873  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 154 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.231889 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 155 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 155 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.231948  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:10.232086  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 154 FOLLOWER]: Advancing to term 155
I1014 19:26:10.231729 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 155 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.233789  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.234102 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 155: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.234304  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 155 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.234463  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 155 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.234935 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 155 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 155: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.235077 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost.
I1014 19:26:10.234973 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 155 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.235080 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 155 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.235285 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 155 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.235298 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 155: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.235611  4555 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:10.235500  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 154 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.235685 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 155 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 155 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 155: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.235720  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.235738  4555 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I1014 19:26:10.236140 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 155 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 155 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.236243 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 155 election: Election decided. Result: candidate lost.
I1014 19:26:10.236313 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 154. Current term is 155. Ops: []
I1014 19:26:10.236621  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:10.236687  3829 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Leader election lost for term 155. Reason: could not achieve majority
I1014 19:26:10.236748  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 154.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I1014 19:26:10.236950  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Stepping down as leader of term 154
I1014 19:26:10.237002  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.237182  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 154.96, Last appended by leader: 96, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.237370  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:10.237429  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:10.237582  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 154 FOLLOWER]: Advancing to term 155
I1014 19:26:10.325057  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 155 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:10.325163  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:10.325235  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 155 FOLLOWER]: Advancing to term 156
I1014 19:26:10.326802  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.327280  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 156 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.327463  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 156 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.328197 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 156 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.328220 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 156 candidate_status { last_received { term: 154 index: 96 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.328382 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 155 FOLLOWER]: Advancing to term 156
I1014 19:26:10.328382 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 155 FOLLOWER]: Advancing to term 156
I1014 19:26:10.328500 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.220s (vote granted)
I1014 19:26:10.328485 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:10.330647 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 156.
I1014 19:26:10.330703 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 156.
I1014 19:26:10.331058 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 156 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.331130 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 156 election: Election decided. Result: candidate won.
I1014 19:26:10.331271 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 156 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.331477  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:10.331552  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 FOLLOWER]: Leader election won for term 156
I1014 19:26:10.331642  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.331845  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 154.96, Last appended by leader: 96, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.332860  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.333055  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.429791 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 156 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 154 index: 96. Preceding OpId from leader: term: 156 index: 97. (index mismatch)
I1014 19:26:10.430294  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I1014 19:26:10.431689  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.431928  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.432904  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.433185  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.433279  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.433564  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.433739 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 156 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 154 index: 96. Preceding OpId from leader: term: 156 index: 97. (index mismatch)
I1014 19:26:10.434247  4942 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I1014 19:26:10.435783  4555 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.436045  4555 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.435940  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.436255  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.533623  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 156 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.533732  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:10.533820  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 156 FOLLOWER]: Advancing to term 157
I1014 19:26:10.535337  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.535851  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 157 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.535969  4943 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 156 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.536026  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 157 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.536061  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:10.536128  4943 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 156 FOLLOWER]: Advancing to term 157
I1014 19:26:10.536578 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 157 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.536787 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 157 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.536753 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 157 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.537150 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 157 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 157 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.538923  4943 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.539297 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 157: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.539502  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 157 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.539721  4942 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 156 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.539994  4942 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.540149  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 157 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.540201 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 157 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 157: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.540174 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 157 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.540313 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 157 election: Election decided. Result: candidate lost.
I1014 19:26:10.540330 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 157 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.540516 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 156. Current term is 157. Ops: []
I1014 19:26:10.540729 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 157 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 157 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.540829 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 156. Current term is 157. Ops: []
I1014 19:26:10.541092 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 157 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.541286 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 157: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.541395  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:10.541496  3829 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Leader election lost for term 157. Reason: could not achieve majority
I1014 19:26:10.541708  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 156.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I1014 19:26:10.541769 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 157 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 157: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.541843 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 157 election: Election decided. Result: candidate lost.
I1014 19:26:10.541916  4942 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 156.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I1014 19:26:10.542150  4410 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Stepping down as leader of term 156
I1014 19:26:10.542213  4410 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.542385  4410 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 156.97, Last appended by leader: 97, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.542642  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:10.542699  4410 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:10.542827  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 156 FOLLOWER]: Advancing to term 157
I1014 19:26:10.544488  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:10.544559  4943 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Leader election lost for term 157. Reason: could not achieve majority
I1014 19:26:10.653632  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 157 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:10.653753  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:10.653831  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 157 FOLLOWER]: Advancing to term 158
I1014 19:26:10.656066  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.656602  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 158 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.656829  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 158 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.657326 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 158 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.657562 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 157 FOLLOWER]: Advancing to term 158
I1014 19:26:10.657537 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 158 candidate_status { last_received { term: 156 index: 97 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.657647 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:10.657708 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 157 FOLLOWER]: Advancing to term 158
I1014 19:26:10.657791 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.180s (vote granted)
I1014 19:26:10.659759 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 158.
I1014 19:26:10.659929 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 158.
I1014 19:26:10.660241 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 158 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.660348 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 158 election: Election decided. Result: candidate won.
I1014 19:26:10.660485 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 158 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.660667  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:10.660771  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 FOLLOWER]: Leader election won for term 158
I1014 19:26:10.660926  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.661154  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 156.97, Last appended by leader: 97, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.662276  4942 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.662467  4942 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.763062 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 158 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 156 index: 97. Preceding OpId from leader: term: 158 index: 98. (index mismatch)
I1014 19:26:10.763602  4942 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I1014 19:26:10.765378  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.765635  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.765880  4942 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.766217  4942 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:10.766631 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 158 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 156 index: 97. Preceding OpId from leader: term: 158 index: 98. (index mismatch)
I1014 19:26:10.767133  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.767349  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.767280  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I1014 19:26:10.769435  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.769342  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.769686  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:10.769727  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:10.831835 32481 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.123s	user 0.002s	sys 0.000s
I1014 19:26:10.831925 32481 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:10.833182 32481 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:10.834579 32481 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 152 vs 158: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.834985 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:10.835297 32481 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.855325  4942 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 158 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:10.855595  4942 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:10.872897  4371 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.503s	user 0.002s	sys 0.000s
I1014 19:26:10.872990  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:10.874054  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:10.881120  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:10.883188  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:10.979426  3829 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 158 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.979462  4943 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 158 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:10.979568  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:10.979557  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:10.979635  4943 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 158 FOLLOWER]: Advancing to term 159
I1014 19:26:10.979645  3829 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 158 FOLLOWER]: Advancing to term 159
I1014 19:26:10.982031  3829 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.982031  4943 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.982594  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 159 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.982631  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 159 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:10.982882 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 158. Current term is 159. Ops: []
I1014 19:26:10.982883  3829 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 159 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:10.982980  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 159 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:10.983405 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 159 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.983631 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 159 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.983597 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 159 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:10.983772 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 159: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.984156 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 159 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 159: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:10.984136  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 158.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I1014 19:26:10.984077 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 159 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:10.984277 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 159 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 159 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.984359 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost.
I1014 19:26:10.984441 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 159 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.984659  3829 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:10.984733  3829 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I1014 19:26:10.984820 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 159 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 159 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:10.984810 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 159 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:10.984953 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 159: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.984941  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Stepping down as leader of term 158
I1014 19:26:10.984985  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:10.985257 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 159 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 159: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:10.985167  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 158.99, Last appended by leader: 99, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:10.985334 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost.
I1014 19:26:10.985339  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:10.985396  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:10.985563  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 158 FOLLOWER]: Advancing to term 159
I1014 19:26:10.985566  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:10.985633  4943 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I1014 19:26:11.078696  4647 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 159 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:11.078788  4647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:11.078863  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 159 FOLLOWER]: Advancing to term 160
I1014 19:26:11.080413  4647 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.080865  4647 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 160 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.081035  4647 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 160 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.081614 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 160 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.081678 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 160 candidate_status { last_received { term: 158 index: 99 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:11.081787 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 159 FOLLOWER]: Advancing to term 160
I1014 19:26:11.081867 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.154s (vote granted)
I1014 19:26:11.081859 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 159 FOLLOWER]: Advancing to term 160
I1014 19:26:11.081967 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.156s (vote granted)
I1014 19:26:11.083271 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 160.
I1014 19:26:11.083617 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 160 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.083685 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 160 election: Election decided. Result: candidate won.
I1014 19:26:11.083925  4647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:11.083998  4647 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 FOLLOWER]: Leader election won for term 160
I1014 19:26:11.084127 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 160.
I1014 19:26:11.084139  4647 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:11.084326  4647 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 158.99, Last appended by leader: 99, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.084583 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 160 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.085299  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.085577  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.184675 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 158 index: 99. Preceding OpId from leader: term: 160 index: 100. (index mismatch)
I1014 19:26:11.185189  4410 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I1014 19:26:11.186580  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.186866  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.187949  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.188148  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.188998 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 160 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 158 index: 99. Preceding OpId from leader: term: 160 index: 100. (index mismatch)
I1014 19:26:11.189447  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.189713  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.189687  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I1014 19:26:11.191401  3829 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.191505  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.191723  3829 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.191823  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.267889  4942 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 160 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.268151  4942 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.456207  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 160 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:11.456308  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:11.456408  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 160 FOLLOWER]: Advancing to term 161
I1014 19:26:11.458914  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.459554  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 161 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.459800  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 161 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.460505 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 161 candidate_status { last_received { term: 160 index: 100 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:11.460841 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 161 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.461030 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 161 candidate_status { last_received { term: 160 index: 100 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:11.461259 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 161 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.461284 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 161 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 161 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.461563 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 160. Current term is 161. Ops: []
I1014 19:26:11.461731 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 161 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 161 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.461895 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 161 election: Election decided. Result: candidate lost.
I1014 19:26:11.462085  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 160.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I1014 19:26:11.462203  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:11.462291  4949 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Leader election lost for term 161. Reason: could not achieve majority
I1014 19:26:11.462366  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 LEADER]: Stepping down as leader of term 160
I1014 19:26:11.462414  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:11.462641  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 160.100, Last appended by leader: 100, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.462818  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:11.462877  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:11.463001  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 160 FOLLOWER]: Advancing to term 161
I1014 19:26:11.463217  4943 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:11.463291  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:11.463366  4943 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 160 FOLLOWER]: Advancing to term 161
I1014 19:26:11.465621  4943 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.466132  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 161 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.466305  4943 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 161 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.467027 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 161 candidate_status { last_received { term: 160 index: 100 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.467221 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 161: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:11.467583 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 161 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 161: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:11.467532 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 161 candidate_status { last_received { term: 160 index: 100 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:11.467712 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:11.469700 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 161.
I1014 19:26:11.470021 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 161 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.470111 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 161 election: Election decided. Result: candidate won.
I1014 19:26:11.470461  4943 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:11.470568  4943 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 FOLLOWER]: Leader election won for term 161
I1014 19:26:11.471048  4943 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:11.471354  4943 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 160.100, Last appended by leader: 100, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.485491  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.485932  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.486748  5087 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:11.489722  5087 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:11.555338 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 160 index: 100. Preceding OpId from leader: term: 161 index: 101. (index mismatch)
I1014 19:26:11.556005  5086 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I1014 19:26:11.557371  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.557642  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.558991  4943 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.559289  4943 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.560811 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 161 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 160 index: 100. Preceding OpId from leader: term: 161 index: 101. (index mismatch)
I1014 19:26:11.560914  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.561182  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.562536  4943 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I1014 19:26:11.564312  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.564716  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.564538  4410 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 161 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.564891  4410 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.660902  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:11.660997  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:11.661078  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 161 FOLLOWER]: Advancing to term 162
I1014 19:26:11.663460  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.664048  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 162 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.664247  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 162 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.664826  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 161 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:11.664963  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:11.664960 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 162 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:11.665071  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 161 FOLLOWER]: Advancing to term 162
I1014 19:26:11.665258 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 162 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:11.665473 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 162 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.665947 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 162 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 162 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.667250  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.667660 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 162: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:11.668108  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 162 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.668130 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 162 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 162: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:11.668205 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 162 election: Election decided. Result: candidate lost.
I1014 19:26:11.668757 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 162 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.668949 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 162: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:11.669549 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 162 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 162: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:11.669663  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 162 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.670073  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:11.670173  4949 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Leader election lost for term 162. Reason: could not achieve majority
I1014 19:26:11.670291 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 161. Current term is 162. Ops: []
I1014 19:26:11.670344 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 162 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:11.670564 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 162 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.670946 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 162 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 162 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.671030 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 162 election: Election decided. Result: candidate lost.
I1014 19:26:11.671278  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:11.671373  4410 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Leader election lost for term 162. Reason: could not achieve majority
I1014 19:26:11.671629  5086 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 161.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I1014 19:26:11.671851  5086 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Stepping down as leader of term 161
I1014 19:26:11.671897  5086 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:11.672109  5086 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 161.101, Last appended by leader: 101, Current term: 161, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.672293  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:11.672346  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:11.672475  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 161 FOLLOWER]: Advancing to term 162
I1014 19:26:11.781123  4410 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:11.781229  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.188s (starting election)
I1014 19:26:11.781311  4410 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 162 FOLLOWER]: Advancing to term 163
I1014 19:26:11.783584  4410 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.784126  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 163 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.784934 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 163 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.785100 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 162 FOLLOWER]: Advancing to term 163
I1014 19:26:11.785181 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.221s (vote granted)
I1014 19:26:11.787331 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 163.
I1014 19:26:11.787775  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 162 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:11.787858  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:11.787925  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 162 FOLLOWER]: Advancing to term 163
I1014 19:26:11.788045 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 163 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.788122 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 163 election: Election decided. Result: candidate won.
I1014 19:26:11.788321  4410 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 163 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:11.789032 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 163 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:11.789324  4410 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.168s (election complete)
I1014 19:26:11.789393  4410 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 FOLLOWER]: Leader election won for term 163
I1014 19:26:11.789512  4410 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:11.789723  4410 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 161.101, Last appended by leader: 101, Current term: 163, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.790366  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.790778 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 163: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:11.790926  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 163 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.791110  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 163 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.791332 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 163 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 163: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:11.792091 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 163 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:11.792564 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 163 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.792533  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.792783  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.793133 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 163 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 163 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.794894 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 163 candidate_status { last_received { term: 161 index: 101 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.795106 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 163: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:11.795431 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 163 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 163: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:11.795536 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 163 election: Election decided. Result: candidate lost.
I1014 19:26:11.796627  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:11.796710  5086 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Leader election lost for term 163. Reason: could not achieve majority
I1014 19:26:11.866972 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 161 index: 101. Preceding OpId from leader: term: 163 index: 102. (index mismatch)
I1014 19:26:11.867709  4410 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I1014 19:26:11.869295  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.869659  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.871192 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 161 index: 101. Preceding OpId from leader: term: 163 index: 102. (index mismatch)
I1014 19:26:11.871825  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I1014 19:26:11.873419  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.873742  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.873311  5088 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.873993  5088 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.884083  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.884341  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:11.884743  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.885006  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.973343  5088 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:11.973479  5088 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:11.973559  5088 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 163 FOLLOWER]: Advancing to term 164
I1014 19:26:11.975915  5088 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:11.976588  5088 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 164 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:11.976855  5088 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 164 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:11.977545 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 164 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:11.977551 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 164 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:11.977785 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 164 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.978127 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 164 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.978196 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 164 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 164 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.978492 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 164 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 164 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:11.978574 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 164 election: Election decided. Result: candidate lost.
I1014 19:26:11.978838  5088 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:11.978911  5088 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Leader election lost for term 164. Reason: could not achieve majority
I1014 19:26:11.998687  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 163 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:11.998980  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:11.999132 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 163. Current term is 164. Ops: []
I1014 19:26:11.999799  4647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 163.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I1014 19:26:11.999991  5157 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Stepping down as leader of term 163
I1014 19:26:12.000044  5157 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:12.000253  5157 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 163.102, Last appended by leader: 102, Current term: 163, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.000510  5157 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:12.000577  5157 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:12.000715  5157 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 163 FOLLOWER]: Advancing to term 164
I1014 19:26:12.069437  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:12.069542  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:12.069625  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 163 FOLLOWER]: Advancing to term 164
I1014 19:26:12.071911  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.072486  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 164 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.072990  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 164 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.073287 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 164 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:12.073536 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:12.075763 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 164 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 164.
I1014 19:26:12.076165 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 164 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.076230 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 164 election: Election decided. Result: candidate won.
I1014 19:26:12.076941  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:12.077149 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 164 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.077371 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 164: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:12.077859 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 164 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 164: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:12.078116  4949 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 FOLLOWER]: Leader election won for term 164
I1014 19:26:12.078295  4949 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:12.078670  4949 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 163.102, Last appended by leader: 102, Current term: 164, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.080417  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 164 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.080775  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.087797  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:12.090464  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:12.129217  5088 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:12.129321  5088 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:12.129418  5088 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 164 FOLLOWER]: Advancing to term 165
I1014 19:26:12.131708  5088 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.132220  5088 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 165 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.132403  5088 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 165 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.133081 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 165 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:12.133103 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 165 candidate_status { last_received { term: 163 index: 102 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:12.133285 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 165 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.133260 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 164 FOLLOWER]: Advancing to term 165
I1014 19:26:12.133347 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:12.135599 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 165 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 165.
I1014 19:26:12.135996 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 165 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.136080 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 165 election: Election decided. Result: candidate won.
I1014 19:26:12.136338  5088 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.186s (election complete)
I1014 19:26:12.136437  5088 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 FOLLOWER]: Leader election won for term 165
I1014 19:26:12.136433 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 165 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 165 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.136991  5088 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:12.137226  5088 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 163.102, Last appended by leader: 102, Current term: 165, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.138239  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.138429  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.175745 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 164. Current term is 165. Ops: []
I1014 19:26:12.176396  5180 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I1014 19:26:12.176769  5180 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 165.103 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 102
I1014 19:26:12.176880  4949 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 LEADER]: Stepping down as leader of term 164
I1014 19:26:12.176937  4949 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:12.177156  4949 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 164.103, Last appended by leader: 103, Current term: 164, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.177366  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:12.177450  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:12.177578  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 164 FOLLOWER]: Advancing to term 165
I1014 19:26:12.251737 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 164 index: 103. Preceding OpId from leader: term: 165 index: 103. (term mismatch)
I1014 19:26:12.251814 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 102
I1014 19:26:12.251857 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 164.103
I1014 19:26:12.251912 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:12.252524  5086 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I1014 19:26:12.254106  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.254381  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.255683  5088 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.255957  5088 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.256608  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.256839  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.256906 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 165 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 163 index: 102. Preceding OpId from leader: term: 165 index: 103. (index mismatch)
I1014 19:26:12.258880  5088 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I1014 19:26:12.260879  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.261147  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.261015  5157 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.261303  5157 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:12.299341  4371 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.416s	user 0.002s	sys 0.000s
I1014 19:26:12.299423  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:12.300567  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:12.301791  4371 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 158 vs 165: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.302109  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.302384  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.302783  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.303186  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.303588  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.304008  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.304388  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.304814  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.305214  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.305639  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.306437  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.311102  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.311559  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.311838  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.312113  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.361090  5157 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 165 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:12.361197  5157 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:12.361280  5157 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 165 FOLLOWER]: Advancing to term 166
I1014 19:26:12.362960  5157 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.363389  5157 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 166 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.363592  5157 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 166 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.364259 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 166 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:12.364336 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 166 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.364461 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 166 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.364568 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 166 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.364909 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 166 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 166 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.365029 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 166 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 166 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.365128 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 166 election: Election decided. Result: candidate lost.
I1014 19:26:12.365411  5157 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:12.365514  5157 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Leader election lost for term 166. Reason: could not achieve majority
I1014 19:26:12.383484  5088 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 165 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.383955  5088 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.384076 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 165. Current term is 166. Ops: []
I1014 19:26:12.384626  5086 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 165.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I1014 19:26:12.384871  5086 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Stepping down as leader of term 165
I1014 19:26:12.384925  5086 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:12.385149  5086 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 165.103, Last appended by leader: 103, Current term: 165, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.385368  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:12.385447  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:12.385599  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 165 FOLLOWER]: Advancing to term 166
I1014 19:26:12.452980  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:12.453079  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (starting election)
I1014 19:26:12.453169  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 165 FOLLOWER]: Advancing to term 166
I1014 19:26:12.454771  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.455199  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 166 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.455374  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 166 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.455993 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 166 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:12.456053 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 166 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.456169 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 166: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:12.456240 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:12.456522 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 166 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 166: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:12.458283 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 166 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 166.
I1014 19:26:12.458627 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 166 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.458696 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 166 election: Election decided. Result: candidate won.
I1014 19:26:12.458950  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:12.459029  4949 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 FOLLOWER]: Leader election won for term 166
I1014 19:26:12.459131  4949 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:12.459381  4949 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 165.103, Last appended by leader: 103, Current term: 166, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.460310  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 166 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.460664  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.496239  5157 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:12.496361  5157 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:12.496451  5157 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 166 FOLLOWER]: Advancing to term 167
I1014 19:26:12.498772  5157 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.499394  5157 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 167 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.499639  5157 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 167 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.500192 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 167 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:12.500232 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 167 candidate_status { last_received { term: 165 index: 103 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.500366 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 167 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.500406 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 166 FOLLOWER]: Advancing to term 167
I1014 19:26:12.500510 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (vote granted)
I1014 19:26:12.500725 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 167 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 167 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.502667 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 167 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 167.
I1014 19:26:12.502972 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 167 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.503039 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 167 election: Election decided. Result: candidate won.
I1014 19:26:12.503273  5157 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:12.503350  5157 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 FOLLOWER]: Leader election won for term 167
I1014 19:26:12.503579  5157 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:12.503789  5157 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 165.103, Last appended by leader: 103, Current term: 167, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.505098  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.505307  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.505535  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:12.507061  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:12.575700 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 166. Current term is 167. Ops: []
I1014 19:26:12.576388  5180 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I1014 19:26:12.576701  5180 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 167.104 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 103
I1014 19:26:12.576843  4949 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 LEADER]: Stepping down as leader of term 166
I1014 19:26:12.576885 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 167 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 166. Current term is 167. Ops: []
I1014 19:26:12.576897  4949 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:12.577112  4949 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 166.104, Last appended by leader: 104, Current term: 166, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.577325  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:12.577394  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:12.577527  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 166 FOLLOWER]: Advancing to term 167
I1014 19:26:12.598600 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 167 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 165 index: 103. Preceding OpId from leader: term: 167 index: 104. (index mismatch)
I1014 19:26:12.599125  5157 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I1014 19:26:12.600433  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.600790  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.602283  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.602519  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.602746 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 167 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 166 index: 104. Preceding OpId from leader: term: 167 index: 104. (term mismatch)
I1014 19:26:12.602811 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 103
I1014 19:26:12.602855 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 166.104
I1014 19:26:12.602898 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 167 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:12.603461  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I1014 19:26:12.603531  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.603716  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.604964  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.605167  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.605790  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.605938  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:12.703438  5087 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.214s	user 0.001s	sys 0.000s
I1014 19:26:12.703522  5087 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:12.703943  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 167 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:12.704064  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:12.704172  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 167 FOLLOWER]: Advancing to term 168
I1014 19:26:12.704809  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 167 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:12.704879  5087 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:12.704900  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:12.704977  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 167 FOLLOWER]: Advancing to term 168
I1014 19:26:12.706437  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.706693  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.707011  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 168 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.707170  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 168 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:12.707249  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 168 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.707429  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 168 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.707893 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 168 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:12.707880 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 168 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.708088 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 168: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:12.708113 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 168 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.708210 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 168 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:12.708379 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 168: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:12.708304 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 168 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:12.708499 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 168 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.708503  5087 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 161 vs 168: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 168 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:12.708856  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 168 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.708909 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 168 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 168: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:12.708945 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 168 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 168 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.709182 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 168 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 168 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:12.709267 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 168 election: Election decided. Result: candidate lost.
W1014 19:26:12.709369  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 168 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.709725 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 168 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 168: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:12.709805 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 168 election: Election decided. Result: candidate lost.
W1014 19:26:12.709828  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 168 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.709831  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:12.709930  5086 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Leader election lost for term 168. Reason: could not achieve majority
I1014 19:26:12.710121  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
W1014 19:26:12.710157  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 168 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.710189  4949 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Leader election lost for term 168. Reason: could not achieve majority
I1014 19:26:12.722046  5157 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 167 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.722322  5157 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.723136 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 167. Current term is 168. Ops: []
I1014 19:26:12.723888  5304 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 167.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I1014 19:26:12.724179  4647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Stepping down as leader of term 167
I1014 19:26:12.724272  4647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:12.724519  4647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 167.104, Last appended by leader: 104, Current term: 167, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.724676  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:12.724732  4647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:12.724730 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 167. Current term is 168. Ops: []
I1014 19:26:12.724845  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 167 FOLLOWER]: Advancing to term 168
I1014 19:26:12.817395  4647 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 168 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:12.817525  4647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:12.817606  4647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 168 FOLLOWER]: Advancing to term 169
I1014 19:26:12.819885  4647 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.820472  4647 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 169 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.820708  4647 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 169 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.821310 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 169 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:12.821357 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 169 candidate_status { last_received { term: 167 index: 104 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:12.821475 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 168 FOLLOWER]: Advancing to term 169
I1014 19:26:12.821521 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 168 FOLLOWER]: Advancing to term 169
I1014 19:26:12.821590 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.158s (vote granted)
I1014 19:26:12.821586 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:12.823019 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 169 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 169.
I1014 19:26:12.823375 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 169 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:12.823453 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 169 election: Election decided. Result: candidate won.
I1014 19:26:12.823698  4647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:12.823722 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 169.
I1014 19:26:12.823776  4647 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 FOLLOWER]: Leader election won for term 169
I1014 19:26:12.823882  4647 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:12.824074 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 169 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:12.824156  4647 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 167.104, Last appended by leader: 104, Current term: 169, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:12.825111  5157 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.825292  5157 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.904702 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 167 index: 104. Preceding OpId from leader: term: 169 index: 105. (index mismatch)
I1014 19:26:12.904711 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 169 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 167 index: 104. Preceding OpId from leader: term: 169 index: 105. (index mismatch)
I1014 19:26:12.905308  5157 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I1014 19:26:12.905578  4647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I1014 19:26:12.906749  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.906987  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.906965  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.907235  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.907939  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.908936  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.909137  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.909190  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.909456  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:12.909467  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:12.909883  4647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:12.910193  4647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.009328  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 169 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:13.009418  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:13.009512  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 169 FOLLOWER]: Advancing to term 170
I1014 19:26:13.011103  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.011544  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 170 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.011749  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 170 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.012362 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 170 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.012573 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 170 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.012902 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 170 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 170 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.013396 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 170 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.013604 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 170 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.013974 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 170 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 170 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.014073 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 170 election: Election decided. Result: candidate lost.
I1014 19:26:13.014292  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:13.014423  5086 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Leader election lost for term 170. Reason: could not achieve majority
I1014 19:26:13.031163 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 169. Current term is 170. Ops: []
I1014 19:26:13.031728  5313 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 169.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I1014 19:26:13.032016  5304 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 LEADER]: Stepping down as leader of term 169
I1014 19:26:13.032076  5304 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:13.032280  5304 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 169.105, Last appended by leader: 105, Current term: 169, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.032513  5304 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:13.032572  5304 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:13.032723  5304 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 169 FOLLOWER]: Advancing to term 170
I1014 19:26:13.060657  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:13.061707  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:13.062870  5182 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 164 vs 169: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.063135  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 169 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.099925  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:13.100011  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:13.100100  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 169 FOLLOWER]: Advancing to term 170
I1014 19:26:13.101620  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.102025  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 170 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.102190  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 170 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.102677 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 170 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.102844 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (vote granted)
I1014 19:26:13.102797 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 170 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.103000 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 170: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:13.103399 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 170 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 170: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:13.104471 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 170 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 170.
I1014 19:26:13.104792 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 170 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.104871 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 170 election: Election decided. Result: candidate won.
I1014 19:26:13.105118  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:13.105226  4949 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 FOLLOWER]: Leader election won for term 170
I1014 19:26:13.105381  4949 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:13.105588  4949 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 169.105, Last appended by leader: 105, Current term: 170, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.106452  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 170 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.106706  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.106974  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:13.108968  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:13.149722  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:13.149818  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.157s (starting election)
I1014 19:26:13.149886  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 170 FOLLOWER]: Advancing to term 171
I1014 19:26:13.151496  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.151931  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 171 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.152104  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 171 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.152753 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 171 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.152812 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 171 candidate_status { last_received { term: 169 index: 105 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.152904 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 170 FOLLOWER]: Advancing to term 171
I1014 19:26:13.152993 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 171 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.152992 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (vote granted)
I1014 19:26:13.153337 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 171 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 171 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.154969 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 171 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 171.
I1014 19:26:13.155287 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 171 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.155354 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 171 election: Election decided. Result: candidate won.
I1014 19:26:13.155658  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.219s (election complete)
I1014 19:26:13.155737  5086 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 FOLLOWER]: Leader election won for term 171
I1014 19:26:13.156162  5086 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:13.156507  5086 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 169.105, Last appended by leader: 105, Current term: 171, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.157404  5376 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.157688  5376 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.157905  5087 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:13.159395  5087 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:13.183889 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 171 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 170. Current term is 171. Ops: []
I1014 19:26:13.184482  5180 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I1014 19:26:13.184762  4949 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 LEADER]: Stepping down as leader of term 170
I1014 19:26:13.184828  4949 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:13.185111  4949 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 170.106, Last appended by leader: 106, Current term: 170, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.185339  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:13.185405  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:13.185557  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 170 FOLLOWER]: Advancing to term 171
I1014 19:26:13.244099 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 171 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 169 index: 105. Preceding OpId from leader: term: 171 index: 106. (index mismatch)
I1014 19:26:13.244658  5376 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I1014 19:26:13.245847  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.246224  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.247388  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.247689  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.247998 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 170 index: 106. Preceding OpId from leader: term: 171 index: 106. (term mismatch)
I1014 19:26:13.248069 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 105
I1014 19:26:13.248114 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 170.106
I1014 19:26:13.248181 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:13.248272  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.248764  5376 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I1014 19:26:13.248926  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.249987  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.250515  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.250519  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.250802  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.344866  5380 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 171 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.345108  5380 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.432657  5304 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 171 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:13.432754  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:13.432823  5304 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 171 FOLLOWER]: Advancing to term 172
I1014 19:26:13.434640  5304 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.435103  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 172 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.435333  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 172 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.435858 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 172 candidate_status { last_received { term: 171 index: 106 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.436012 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 172 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.435992 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 172 candidate_status { last_received { term: 171 index: 106 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.436194 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 172 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.436461 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 172 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 172 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.436607 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 172 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 172 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.436707 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 172 election: Election decided. Result: candidate lost.
I1014 19:26:13.436921  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:13.437032  5304 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Leader election lost for term 172. Reason: could not achieve majority
I1014 19:26:13.445866  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:13.445950  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:13.446018  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 171 FOLLOWER]: Advancing to term 172
I1014 19:26:13.447333 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 171. Current term is 172. Ops: []
I1014 19:26:13.447819  5380 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 171.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I1014 19:26:13.448065  5086 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 LEADER]: Stepping down as leader of term 171
I1014 19:26:13.448109  5086 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:13.448177  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.448323  5086 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 171.106, Last appended by leader: 106, Current term: 171, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.448516  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:13.448567  5086 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:13.448668  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 171 FOLLOWER]: Advancing to term 172
I1014 19:26:13.448714  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 172 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.448917  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 172 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.449580 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 172 candidate_status { last_received { term: 171 index: 106 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.449654 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 172 candidate_status { last_received { term: 171 index: 106 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.449821 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 172: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:13.450197 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 172 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 172: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:13.450904 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:13.452960 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 172.
I1014 19:26:13.453325 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 172 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.453393 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 172 election: Election decided. Result: candidate won.
I1014 19:26:13.453722  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:13.453797  4949 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 FOLLOWER]: Leader election won for term 172
I1014 19:26:13.453948  4949 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:13.454145  4949 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 171.106, Last appended by leader: 106, Current term: 172, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.455487  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.455667  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.496038  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:13.497007  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:13.499297 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 171 index: 106. Preceding OpId from leader: term: 172 index: 108. (index mismatch)
I1014 19:26:13.499330 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 171 index: 106. Preceding OpId from leader: term: 172 index: 108. (index mismatch)
I1014 19:26:13.499884  4949 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I1014 19:26:13.500099  5180 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I1014 19:26:13.502358  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.502339  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.502599  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.502605  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.504235  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.504441  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.504652  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:13.506557  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:13.507122  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.507380  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.507932  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.507927  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.508082  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.508136  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:13.581003  4371 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.074s	user 0.002s	sys 0.000s
I1014 19:26:13.581069  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:13.582054  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:13.583170  4371 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 167 vs 172: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.583461  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.583776  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.584127  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.584528  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:13.584781  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 172 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.603662  5304 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:13.603770  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:13.603852  5304 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 172 FOLLOWER]: Advancing to term 173
I1014 19:26:13.605818  5304 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.606392  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 173 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.607165 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 173 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.607378 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 173 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.607373 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 172. Current term is 173. Ops: []
I1014 19:26:13.607937  4949 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 172.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I1014 19:26:13.608002 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 173 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 173 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.608131  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 173 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.608263  5180 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 LEADER]: Stepping down as leader of term 172
I1014 19:26:13.608325  5180 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:13.608566  5180 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 172.108, Last appended by leader: 108, Current term: 172, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.608791  5180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:13.608856  5180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:13.608855 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 173 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.608974  5180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 172 FOLLOWER]: Advancing to term 173
I1014 19:26:13.609051 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 173 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.609483 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 173 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 173 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.609567 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 173 election: Election decided. Result: candidate lost.
I1014 19:26:13.609853  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:13.609933  5304 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Leader election lost for term 173. Reason: could not achieve majority
I1014 19:26:13.684176  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:13.684263  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:13.684334  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 172 FOLLOWER]: Advancing to term 173
I1014 19:26:13.686693  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.687223  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 173 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.687394  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 173 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.688143 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 173 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.688360 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 173: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:13.688539 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 173 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.688717 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:13.688802 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 173 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 173: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:13.690790 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 173 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 173.
I1014 19:26:13.691123 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 173 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.691179 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 173 election: Election decided. Result: candidate won.
I1014 19:26:13.691416  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:13.691504  5086 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 FOLLOWER]: Leader election won for term 173
I1014 19:26:13.691649  5086 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:13.691843  5086 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 172.108, Last appended by leader: 108, Current term: 173, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.692790  5380 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 173 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.693045  5380 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.716676  5304 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:13.716784  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:13.716871  5304 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 173 FOLLOWER]: Advancing to term 174
I1014 19:26:13.718353  5304 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.718850  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 174 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.719074  5304 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 174 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.719650 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 174 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.719839 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 173 FOLLOWER]: Advancing to term 174
I1014 19:26:13.719782 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 174 candidate_status { last_received { term: 172 index: 108 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.719905 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:13.719967 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 174 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.720441 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 174 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 174 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.721302 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 174 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 174.
I1014 19:26:13.721635 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 174 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.721707 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 174 election: Election decided. Result: candidate won.
I1014 19:26:13.721940  5304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:13.722021  5304 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 FOLLOWER]: Leader election won for term 174
I1014 19:26:13.722625  5304 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:13.722970  5304 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 172.108, Last appended by leader: 108, Current term: 174, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.724220  5501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.724454  5501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.724670  4371 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:13.726658  4371 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:13.804988 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 174 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 173. Current term is 174. Ops: []
I1014 19:26:13.805354 29011 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 LEADER]: Stepping down as leader of term 173
I1014 19:26:13.805490 29011 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:13.805721 29011 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 173.109, Last appended by leader: 109, Current term: 173, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.806013 29011 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:13.806082 29011 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:13.806141 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 173 FOLLOWER]: Advancing to term 174
I1014 19:26:13.808590 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 174 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 173 index: 109. Preceding OpId from leader: term: 174 index: 109. (term mismatch)
I1014 19:26:13.808657 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 108
I1014 19:26:13.808697 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 173.109
I1014 19:26:13.808753 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 174 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:13.809578 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 173. Current term is 174. Ops: []
I1014 19:26:13.810072  5501 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I1014 19:26:13.811447  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.811642  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.812499  5501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.812686  5501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.812981 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 174 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 172 index: 108. Preceding OpId from leader: term: 174 index: 109. (index mismatch)
I1014 19:26:13.813315  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.813539  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.813788  5304 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I1014 19:26:13.815119  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.815471  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.815405  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.815670  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:13.913446  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 174 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:13.913544  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:13.913636  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 174 FOLLOWER]: Advancing to term 175
I1014 19:26:13.915395  5180 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 174 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:13.915482  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:13.915586  5180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 174 FOLLOWER]: Advancing to term 175
I1014 19:26:13.915988  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.916527  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 175 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.916734  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 175 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:13.917593 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 175 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.917613 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 175 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:13.917795 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 175 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.917861  5180 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.918200 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 175: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:13.918272 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 175 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 175 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.918376  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 175 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:13.918563  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 175 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:13.919183 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 175 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 175: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:13.919270 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 174. Current term is 175. Ops: []
I1014 19:26:13.919332 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 175 election: Election decided. Result: candidate lost.
I1014 19:26:13.919329  5501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 174 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:13.919597  5501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:13.919636  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:13.919754  5086 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Leader election lost for term 175. Reason: could not achieve majority
I1014 19:26:13.919816 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 175 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:13.920001 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 175: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:13.920043  5501 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 174.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I1014 19:26:13.920289 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 175 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:13.920603 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 175 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.920909 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 175 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 175: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:13.921219 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 175 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 175 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:13.921308 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 175 election: Election decided. Result: candidate lost.
I1014 19:26:13.921605  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:13.921676  5180 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Leader election lost for term 175. Reason: could not achieve majority
I1014 19:26:13.926697 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 174. Current term is 175. Ops: []
I1014 19:26:13.927222  5501 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 174.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I1014 19:26:13.937914  5304 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Stepping down as leader of term 174
I1014 19:26:13.938025  5304 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:13.938340  5304 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 174.109, Last appended by leader: 109, Current term: 174, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:13.938599  5304 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:13.938688  5304 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:13.938863  5304 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 174 FOLLOWER]: Advancing to term 175
I1014 19:26:14.023977  5180 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:14.024076  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:14.024152  5180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 175 FOLLOWER]: Advancing to term 176
I1014 19:26:14.026331  5180 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.026899  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 176 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.027127  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 176 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.027669 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 176 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:14.027838 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 175 FOLLOWER]: Advancing to term 176
I1014 19:26:14.027917 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:14.029965 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 176.
I1014 19:26:14.030369 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 176 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.030316 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 176 candidate_status { last_received { term: 174 index: 109 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:14.030478 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 176 election: Election decided. Result: candidate won.
I1014 19:26:14.030524 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 175 FOLLOWER]: Advancing to term 176
I1014 19:26:14.030606 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.150s (vote granted)
I1014 19:26:14.030689  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.179s (election complete)
I1014 19:26:14.030778  5180 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 FOLLOWER]: Leader election won for term 176
I1014 19:26:14.030899  5180 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:14.031111  5180 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 174.109, Last appended by leader: 109, Current term: 176, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.032550  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.032699 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 176 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 176.
I1014 19:26:14.032754  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.033035 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 176 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.118072 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 176 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 174 index: 109. Preceding OpId from leader: term: 176 index: 110. (index mismatch)
I1014 19:26:14.118620  4949 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I1014 19:26:14.120224  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.120478  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.121134  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.121338  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.121649 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 174 index: 109. Preceding OpId from leader: term: 176 index: 110. (index mismatch)
I1014 19:26:14.122216  4949 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I1014 19:26:14.122318  5087 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:14.123358  5087 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:14.123589  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.123821  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.123960  5304 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.124151  5304 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.124678  5087 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 171 vs 176: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:14.125582  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.125581  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.125800  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:14.125982  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:14.126241  5087 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.179433  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:14.180456  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:14.184412  5180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 176 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.184703  5180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.194663  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:14.196467  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:14.231401  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:14.232398  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:14.416846  5086 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 176 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:14.416925  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:14.416995  5086 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 176 FOLLOWER]: Advancing to term 177
I1014 19:26:14.417652 28816 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.20.201.1:45525
I1014 19:26:14.419154  5086 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.419675  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 177 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.419833  5086 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 177 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:14.420550 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 177 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:14.420601 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 177 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:14.420738 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.420790 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.421169 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 177 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.421344 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 177 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.421417 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 177 election: Election decided. Result: candidate lost.
I1014 19:26:14.421676  5086 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:14.421741  5086 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Leader election lost for term 177. Reason: could not achieve majority
I1014 19:26:14.428539  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:14.428634  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:14.428699  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 176 FOLLOWER]: Advancing to term 177
I1014 19:26:14.430958  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 177 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.431458  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 177 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:14.432581 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 176. Current term is 177. Ops: []
I1014 19:26:14.432631 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 177 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:14.432812 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.433195  4949 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 176.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I1014 19:26:14.433331 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 177 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 177 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.433434  5180 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Stepping down as leader of term 176
I1014 19:26:14.433490  5180 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:14.433523  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 177 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.433704  5180 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 176.112, Last appended by leader: 112, Current term: 176, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.433910  5180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:14.433979  5180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:14.434103  5180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 176 FOLLOWER]: Advancing to term 177
I1014 19:26:14.434041 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 177 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:14.434231 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 177: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:14.434610 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 177 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 177: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:14.434687 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 177 election: Election decided. Result: candidate lost.
I1014 19:26:14.434926  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:26:14.434994  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 177 FOLLOWER]: Leader election lost for term 177. Reason: could not achieve majority
I1014 19:26:14.530290  5180 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 177 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:14.530397  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:14.530483  5180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 177 FOLLOWER]: Advancing to term 178
I1014 19:26:14.532701  5180 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.533232  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 178 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.533465  5180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 178 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.534538 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 178 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:14.534549 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 178 candidate_status { last_received { term: 176 index: 112 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:14.534724 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 177 FOLLOWER]: Advancing to term 178
I1014 19:26:14.534725 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 177 FOLLOWER]: Advancing to term 178
I1014 19:26:14.534807 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.215s (vote granted)
I1014 19:26:14.534817 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (vote granted)
I1014 19:26:14.536952 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 178 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 178.
I1014 19:26:14.537189 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 178 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 178.
I1014 19:26:14.537676 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 178 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.537753 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 178 election: Election decided. Result: candidate won.
I1014 19:26:14.537972  5180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:14.538085  5180 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 FOLLOWER]: Leader election won for term 178
I1014 19:26:14.538170 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 178 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.538272  5180 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:14.538482  5180 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 176.112, Last appended by leader: 112, Current term: 178, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.539793  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.540000  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.540216  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:14.542233  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:14.641242 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 178 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 176 index: 112. Preceding OpId from leader: term: 178 index: 113. (index mismatch)
I1014 19:26:14.641903  4949 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I1014 19:26:14.643326  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.643631  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.645160  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.645342  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.645561 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 178 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 176 index: 112. Preceding OpId from leader: term: 178 index: 113. (index mismatch)
I1014 19:26:14.645046  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.645824  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.646348  4949 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I1014 19:26:14.662044  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.662317  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.662936  5787 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.663179  5787 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.745638  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 178 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:14.745745  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:14.745831  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 178 FOLLOWER]: Advancing to term 179
I1014 19:26:14.747962  5787 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 178 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:14.748052  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:14.748126  5787 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 178 FOLLOWER]: Advancing to term 179
I1014 19:26:14.748106  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.748661  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 179 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:14.749418 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 179 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:14.749655 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 179 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.750095 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 179 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 179 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.750212  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 179 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.750247  5787 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.750720  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 179 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.750885  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 179 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:14.751505 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 179 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:14.751696 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 179: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:14.751729 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 179 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:14.751718 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 179 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:14.751880 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 179 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.751891 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 179: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:14.752240 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 179 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 179 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:14.752292 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 179 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 179: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:14.752373 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 179 election: Election decided. Result: candidate lost.
I1014 19:26:14.752637  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:14.752741  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Leader election lost for term 179. Reason: could not achieve majority
I1014 19:26:14.752799 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 179 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 179: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:14.752908 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 179 election: Election decided. Result: candidate lost.
I1014 19:26:14.753149  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:14.753238  5787 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Leader election lost for term 179. Reason: could not achieve majority
I1014 19:26:14.762069  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 178 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.762362  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.762423 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 178. Current term is 179. Ops: []
I1014 19:26:14.762956 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 178. Current term is 179. Ops: []
I1014 19:26:14.762959  5644 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 178.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I1014 19:26:14.763231  5786 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Stepping down as leader of term 178
I1014 19:26:14.763298  5786 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:14.763528  5786 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 178.113, Last appended by leader: 113, Current term: 178, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.763741  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:14.763794  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:14.763893  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 178 FOLLOWER]: Advancing to term 179
I1014 19:26:14.855696  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 179 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:14.855793  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (starting election)
I1014 19:26:14.855859  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 179 FOLLOWER]: Advancing to term 180
I1014 19:26:14.858031  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.858556  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 180 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.858731  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 180 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.859397 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 180 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:14.859405 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 180 candidate_status { last_received { term: 178 index: 113 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:14.859611 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 179 FOLLOWER]: Advancing to term 180
I1014 19:26:14.859621 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 179 FOLLOWER]: Advancing to term 180
I1014 19:26:14.859705 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.201s (vote granted)
I1014 19:26:14.859706 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.168s (vote granted)
I1014 19:26:14.861938 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 180 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 180.
I1014 19:26:14.862221 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 180.
I1014 19:26:14.862416 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 180 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:14.862512 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 180 election: Election decided. Result: candidate won.
I1014 19:26:14.862684 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 180 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:14.862906  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (election complete)
I1014 19:26:14.862970  5786 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 FOLLOWER]: Leader election won for term 180
I1014 19:26:14.863106  5786 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:14.863343  5786 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 178.113, Last appended by leader: 113, Current term: 180, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:14.864529  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.864781  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.956985 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 180 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 178 index: 113. Preceding OpId from leader: term: 180 index: 114. (index mismatch)
I1014 19:26:14.957572  5644 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I1014 19:26:14.958827  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.959110  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.960187  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.960378  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:14.960602  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.960788  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.961123 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 178 index: 113. Preceding OpId from leader: term: 180 index: 114. (index mismatch)
I1014 19:26:14.961923  5786 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I1014 19:26:14.963166  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.963425  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:14.963811  5787 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:14.964018  5787 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.048784  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 180 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.049049  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.061435  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 180 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:15.061558  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:15.061648  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 180 FOLLOWER]: Advancing to term 181
I1014 19:26:15.063874  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.064416  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 181 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.065246 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 181 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.065469 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 181 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.065990 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 181 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 181 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.066151  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 181 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.066843 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 181 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.067064 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 181 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.067584 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 181 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 181 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.067661 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 181 election: Election decided. Result: candidate lost.
I1014 19:26:15.067978  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:15.068080  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Leader election lost for term 181. Reason: could not achieve majority
I1014 19:26:15.070112 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 180. Current term is 181. Ops: []
I1014 19:26:15.070696  5786 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 180.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I1014 19:26:15.070940  4949 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 LEADER]: Stepping down as leader of term 180
I1014 19:26:15.071012  4949 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:15.071243  4949 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 180.114, Last appended by leader: 114, Current term: 180, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.071404  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:15.071461  4949 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:15.071637  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 180 FOLLOWER]: Advancing to term 181
I1014 19:26:15.150354  5787 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:15.150457  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:15.150544  5787 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 180 FOLLOWER]: Advancing to term 181
I1014 19:26:15.151443  4949 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 181 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:15.151568  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:15.151648  4949 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 181 FOLLOWER]: Advancing to term 182
I1014 19:26:15.152154  5787 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 181 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.152652  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 181 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.152858  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 181 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.153771  4949 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.153841 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 181 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.154042 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 181: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.154189  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 182 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.154353  4949 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 182 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.154608 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 181 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 181: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.154726 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 181 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.154925 28826 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 181. Current term is 182.
W1014 19:26:15.155298 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 181 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 181. Current term is 182.
I1014 19:26:15.155349 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 181 election: Cancelling election due to peer responding with higher term
I1014 19:26:15.155752 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 182 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.155906 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 181 FOLLOWER]: Advancing to term 182
I1014 19:26:15.155992 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (vote granted)
I1014 19:26:15.157981 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 182 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 182.
I1014 19:26:15.158337 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 182 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.158412 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 182 election: Election decided. Result: candidate won.
I1014 19:26:15.158435 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 182 candidate_status { last_received { term: 180 index: 114 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.158589 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 181 FOLLOWER]: Advancing to term 182
I1014 19:26:15.158664 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.184s (vote granted)
I1014 19:26:15.158773  4949 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:15.158861  4949 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Leader election won for term 182
I1014 19:26:15.158965  4949 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:15.159163  4949 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 180.114, Last appended by leader: 114, Current term: 182, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.160048  4949 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.160274  4949 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.160666 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 182 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 182.
I1014 19:26:15.161047 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 182 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.161989  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:15.162062  5787 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 182 FOLLOWER]: Leader election lost for term 181. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 181. Current term is 182.
I1014 19:26:15.243573 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 182 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 180 index: 114. Preceding OpId from leader: term: 182 index: 115. (index mismatch)
I1014 19:26:15.244096  5786 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I1014 19:26:15.245381  5787 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.245702  5787 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.246852  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.247077  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.247300 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 182 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 180 index: 114. Preceding OpId from leader: term: 182 index: 115. (index mismatch)
I1014 19:26:15.247701  5787 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.247905  5787 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.248013  5786 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I1014 19:26:15.250448  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.250856  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.250267  6008 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.251045  6008 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.274278  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:15.275265  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:15.277721  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 182 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.277951  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.281113  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:15.282927  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:15.381296  6008 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 182 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:15.381391  6008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:15.381491  6008 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 182 FOLLOWER]: Advancing to term 183
I1014 19:26:15.383090  6008 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.383532  6008 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 183 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.383766  6008 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 183 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.383985  5787 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 182 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:15.384079  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:15.384160  5787 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 182 FOLLOWER]: Advancing to term 183
I1014 19:26:15.384245 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 183 candidate_status { last_received { term: 182 index: 116 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.384445 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 183 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.384879 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 183 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 183 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.385619 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 182. Current term is 183. Ops: []
I1014 19:26:15.386245  5644 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 182.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I1014 19:26:15.386497  5644 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 LEADER]: Stepping down as leader of term 182
I1014 19:26:15.386584  5644 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:15.386617  5787 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.386785  5644 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 182.116, Last appended by leader: 116, Current term: 182, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.386993  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:15.386978 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 182. Current term is 183. Ops: []
I1014 19:26:15.387079  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:15.387197  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 182 FOLLOWER]: Advancing to term 183
I1014 19:26:15.387221 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 183 candidate_status { last_received { term: 182 index: 116 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.387452 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 183: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:15.387887  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 183 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.388044  5787 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 183 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.391166 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 183 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 183: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:15.391263 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 183 election: Election decided. Result: candidate lost.
I1014 19:26:15.391221 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 183 candidate_status { last_received { term: 182 index: 116 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.391396 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:15.391542  6008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:15.391646  6008 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Leader election lost for term 183. Reason: could not achieve majority
I1014 19:26:15.391916 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 183 candidate_status { last_received { term: 182 index: 116 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.392093 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 183: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.392552 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 183 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 183: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.393277 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 183 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 183.
I1014 19:26:15.393762 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 183 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.393863 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 183 election: Election decided. Result: candidate won.
I1014 19:26:15.394132  5787 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:15.394207  5787 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Leader election won for term 183
I1014 19:26:15.394323  5787 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:15.394583  5787 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 182.116, Last appended by leader: 116, Current term: 183, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.395750  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.396064  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.397032  6109 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:15.399549  6109 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:15.420245 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
W1014 19:26:15.436745  4371 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.710s	user 0.002s	sys 0.000s
I1014 19:26:15.436820  4371 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:15.437811  4371 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:15.438982  4371 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 174 vs 183: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 183 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:15.439281  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 183 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:15.439539  4371 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 183 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.483966 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 183 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 182 index: 116. Preceding OpId from leader: term: 183 index: 117. (index mismatch)
I1014 19:26:15.484570  5086 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I1014 19:26:15.485967  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.486263  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.487015  5787 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.487360  5787 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.487557  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.487771  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.488334 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 182 index: 116. Preceding OpId from leader: term: 183 index: 117. (index mismatch)
I1014 19:26:15.488821  6110 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I1014 19:26:15.490232  6008 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.490103  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 183 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.490569  6008 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.490658  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.587708  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 183 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:15.587801  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:15.587872  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 183 FOLLOWER]: Advancing to term 184
I1014 19:26:15.589707  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.590255  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 184 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.590458  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 184 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.591392  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:15.591506  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:15.591578  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 183 FOLLOWER]: Advancing to term 184
I1014 19:26:15.591392 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 184 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.591989 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 184 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.592202 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 184 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.592658 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 184 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 184 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.593675  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 184 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.594034 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 184 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 184: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.594188  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 184 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.594444 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 184 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 184 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 184: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.594565 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 184 election: Election decided. Result: candidate lost.
I1014 19:26:15.594796  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 184 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.594805 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 184 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.595130 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 184: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:15.595346 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 183. Current term is 184. Ops: []
I1014 19:26:15.595397 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 184 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.595541 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 184 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 184: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:15.595583 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 184 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.595667  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:15.595803  5786 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Leader election lost for term 184. Reason: could not achieve majority
I1014 19:26:15.595937 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 184 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 184 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.596012 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 184 election: Election decided. Result: candidate lost.
I1014 19:26:15.596165  6110 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 183.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I1014 19:26:15.596252  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:15.596318  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 184 FOLLOWER]: Leader election lost for term 184. Reason: could not achieve majority
I1014 19:26:15.596390  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Stepping down as leader of term 183
I1014 19:26:15.596447  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:15.596704  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 183.117, Last appended by leader: 117, Current term: 183, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.596873  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:15.596945  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:15.597084  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 183 FOLLOWER]: Advancing to term 184
I1014 19:26:15.683408  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 184 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:15.683535  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:15.683615  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 184 FOLLOWER]: Advancing to term 185
I1014 19:26:15.685271  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.685775  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 185 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.685992  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 185 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.686496 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 185 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.686666 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 184 FOLLOWER]: Advancing to term 185
I1014 19:26:15.686749 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.162s (vote granted)
I1014 19:26:15.686697 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 185 candidate_status { last_received { term: 183 index: 117 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.686894 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 184 FOLLOWER]: Advancing to term 185
I1014 19:26:15.686976 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.184s (vote granted)
I1014 19:26:15.688207 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 185 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 185.
I1014 19:26:15.688547 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 185 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.688619 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 185 election: Election decided. Result: candidate won.
I1014 19:26:15.688854  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:15.688931  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 FOLLOWER]: Leader election won for term 185
I1014 19:26:15.689071 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 185.
I1014 19:26:15.689075  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:15.689255  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 183.117, Last appended by leader: 117, Current term: 185, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.689540 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 185 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.690379  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.690577  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.773023 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 185 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 183 index: 117. Preceding OpId from leader: term: 185 index: 118. (index mismatch)
I1014 19:26:15.773571  5086 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I1014 19:26:15.774963  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.775257  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.776348  5086 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.776638  5086 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.776877 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 183 index: 117. Preceding OpId from leader: term: 185 index: 118. (index mismatch)
I1014 19:26:15.776876  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.777091  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.777632  5787 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I1014 19:26:15.779158  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.779490  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.779731  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.779973  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:15.862056  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 185 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.862378  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:15.877322  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 185 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:15.877413  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:15.877511  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 185 FOLLOWER]: Advancing to term 186
I1014 19:26:15.879039  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.879540  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 186 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:15.879783  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 186 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.880286 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 186 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:15.880492 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 186 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.880475 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 186 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.880692 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 186 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.880888 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 186 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 186 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.881130 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 186 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 186 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:15.881222 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 186 election: Election decided. Result: candidate lost.
I1014 19:26:15.881494  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:15.881593  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Leader election lost for term 186. Reason: could not achieve majority
I1014 19:26:15.899883 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 185. Current term is 186. Ops: []
I1014 19:26:15.900451  6110 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 185.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I1014 19:26:15.900729  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 LEADER]: Stepping down as leader of term 185
I1014 19:26:15.900787  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:15.900985  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 185.118, Last appended by leader: 118, Current term: 185, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.901188  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:15.901238  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:15.901427  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 185 FOLLOWER]: Advancing to term 186
I1014 19:26:15.963683  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:15.963775  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:15.963848  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 185 FOLLOWER]: Advancing to term 186
I1014 19:26:15.966011  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.966540  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 186 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:15.966975  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 186 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.967249 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 186 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:15.967437 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 186: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.968235 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 186 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 186: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:15.968492 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 186 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:15.968693 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:15.970774 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 186 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 186.
I1014 19:26:15.971101 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 186 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:15.971166 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 186 election: Election decided. Result: candidate won.
I1014 19:26:15.971500  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:15.971621  5786 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 FOLLOWER]: Leader election won for term 186
I1014 19:26:15.971858  5786 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:15.972084  5786 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 185.118, Last appended by leader: 118, Current term: 186, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:15.973059  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 186 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:15.973345  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.017520  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:16.017660  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.170s (starting election)
I1014 19:26:16.017796  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 186 FOLLOWER]: Advancing to term 187
I1014 19:26:16.019780  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.020258  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 187 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:16.020478  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 187 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.021056 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 187 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:16.021237 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 187 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.021356 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 187 candidate_status { last_received { term: 185 index: 118 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.021555 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 186 FOLLOWER]: Advancing to term 187
I1014 19:26:16.021625 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 187 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 187 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.021644 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:16.023720 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 187.
I1014 19:26:16.024121 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 187 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.024211 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 187 election: Election decided. Result: candidate won.
I1014 19:26:16.024502  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.197s (election complete)
I1014 19:26:16.024603  5596 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 FOLLOWER]: Leader election won for term 187
I1014 19:26:16.025022  5596 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:16.025317  5596 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 185.118, Last appended by leader: 118, Current term: 187, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.026432  6363 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.026651  6363 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.027339  6365 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:16.029299  6365 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:16.061543 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 186. Current term is 187. Ops: []
I1014 19:26:16.062168  5786 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I1014 19:26:16.062422  5786 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 187.119 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 118
I1014 19:26:16.062557  5786 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 LEADER]: Stepping down as leader of term 186
I1014 19:26:16.062611  5786 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:16.062794  5786 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 186.119, Last appended by leader: 119, Current term: 186, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.062965  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:16.063028  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:16.063144  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 186 FOLLOWER]: Advancing to term 187
I1014 19:26:16.122104 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 185 index: 118. Preceding OpId from leader: term: 187 index: 119. (index mismatch)
I1014 19:26:16.122740  6363 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I1014 19:26:16.124088  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.124366  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.126863  6363 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.127099  6363 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.128171 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 187 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 186 index: 119. Preceding OpId from leader: term: 187 index: 119. (term mismatch)
I1014 19:26:16.128229 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 118
I1014 19:26:16.128268 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 186.119
I1014 19:26:16.128316 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 187 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:16.128479  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.128727  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.129367  5596 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I1014 19:26:16.130628  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.130952  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.130774  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.131067  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.227123  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.227378  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.299592  6109 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:16.300539  6109 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:16.301658  6109 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 183 vs 187: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.301942  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.302177  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.302410  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.302677  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 187 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.404296  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 187 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:16.404446  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:16.404567  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 187 FOLLOWER]: Advancing to term 188
W1014 19:26:16.405194  5182 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.122s	user 0.001s	sys 0.001s
I1014 19:26:16.405241  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:16.406227  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:16.408697  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.409584  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 188 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.410454 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 187. Current term is 188. Ops: []
I1014 19:26:16.410405 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 188 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:16.410614 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 188 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.411013  5596 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 187.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I1014 19:26:16.411072 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 188 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 188 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.411185  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 188 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.411250  5596 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Stepping down as leader of term 187
I1014 19:26:16.411301  5596 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:16.411520  5596 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 187.119, Last appended by leader: 119, Current term: 187, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.411732  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:16.411798  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:16.411944  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 187 FOLLOWER]: Advancing to term 188
I1014 19:26:16.411969  5182 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 182 vs 188: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.412294  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.412617  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.412885  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.413153  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.413422  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.413707  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.413986  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.414285  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 188 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.414639 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 188 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.414857 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 188 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.415227 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 188 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 188 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.415315 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 188 election: Election decided. Result: candidate lost.
I1014 19:26:16.415597  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:16.415668  5786 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Leader election lost for term 188. Reason: could not achieve majority
I1014 19:26:16.486721  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:16.486834  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:16.486908  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 187 FOLLOWER]: Advancing to term 188
I1014 19:26:16.489145  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.489724  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 188 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.489958  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 188 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:16.490700 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 188 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:16.490684 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 188 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:16.490865 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (vote granted)
I1014 19:26:16.490902 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 188: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:16.491281 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 188 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 188: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:16.492913 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 188 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 188.
I1014 19:26:16.493306 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 188 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.493396 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 188 election: Election decided. Result: candidate won.
I1014 19:26:16.493697  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:16.493775  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 FOLLOWER]: Leader election won for term 188
I1014 19:26:16.494369  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:16.494642  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 187.119, Last appended by leader: 119, Current term: 188, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.495694  6507 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 188 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.495990  6507 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.496217  6109 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:16.498210  6109 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:16.561075  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:16.561167  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.155s (starting election)
I1014 19:26:16.561245  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 188 FOLLOWER]: Advancing to term 189
I1014 19:26:16.562853  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.563252  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 189 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.563424  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 189 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.564013 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 189 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:16.564167 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 188 FOLLOWER]: Advancing to term 189
I1014 19:26:16.564126 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 189 candidate_status { last_received { term: 187 index: 119 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.564247 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:16.564350 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 189 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.564757 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 189 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 189 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.565898 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 189 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 189.
I1014 19:26:16.566298 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 189 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.566370 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 189 election: Election decided. Result: candidate won.
I1014 19:26:16.566622  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.197s (election complete)
I1014 19:26:16.566699  5786 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 FOLLOWER]: Leader election won for term 189
I1014 19:26:16.566843  5786 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:16.567042  5786 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 187.119, Last appended by leader: 119, Current term: 189, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.568800  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.568955  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.569157  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:16.570762  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:16.576802 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 188. Current term is 189. Ops: []
I1014 19:26:16.577334  6507 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I1014 19:26:16.577541  6507 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 189.120 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 119
I1014 19:26:16.577658  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 LEADER]: Stepping down as leader of term 188
I1014 19:26:16.577739  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:16.578023  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 188.120, Last appended by leader: 120, Current term: 188, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.578259  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:16.578320  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:16.578452  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 188 FOLLOWER]: Advancing to term 189
I1014 19:26:16.668949 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 189 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 187 index: 119. Preceding OpId from leader: term: 189 index: 120. (index mismatch)
I1014 19:26:16.668988 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 189 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 188 index: 120. Preceding OpId from leader: term: 189 index: 120. (term mismatch)
I1014 19:26:16.669044 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 119
I1014 19:26:16.669070 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 188.120
I1014 19:26:16.669136 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 189 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:16.669461  5786 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I1014 19:26:16.669737  5644 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I1014 19:26:16.670825  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.671025  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.671082  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.671356  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.673574  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.673823  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.674121  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.674268  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.674556  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.674716  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.685909  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 189 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.686084  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.773020  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 189 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:16.773140  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:16.773211  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 189 FOLLOWER]: Advancing to term 190
I1014 19:26:16.774617  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 189 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:16.774705  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:16.774775  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 189 FOLLOWER]: Advancing to term 190
I1014 19:26:16.775588  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 190 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.776195  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 190 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.776444  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 190 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:16.777017 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 190 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:16.776891  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.777535  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 190 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:16.777701  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 190 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.777726 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 190 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:16.777958 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 190 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.778281 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 190 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:16.778522 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 190 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.778548 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 190: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:16.778631 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 190 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.778863 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 190 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 190: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:16.778940 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 190 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 190 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.778924 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 190 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 190 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.779237 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 190 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 190 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 190: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:16.779314 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 190 election: Election decided. Result: candidate lost.
I1014 19:26:16.779595  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:16.779670  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Leader election lost for term 190. Reason: could not achieve majority
I1014 19:26:16.779727 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 190 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 190: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:16.779829 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 190 election: Election decided. Result: candidate lost.
I1014 19:26:16.780061  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:16.780138  6110 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 190 FOLLOWER]: Leader election lost for term 190. Reason: could not achieve majority
I1014 19:26:16.781548 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 189. Current term is 190. Ops: []
I1014 19:26:16.782179  5786 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 189.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I1014 19:26:16.782435  5786 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Stepping down as leader of term 189
I1014 19:26:16.782486  5786 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:16.782737  5786 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 189.120, Last appended by leader: 120, Current term: 189, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.782933  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:16.782994  5786 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:16.783128  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 189 FOLLOWER]: Advancing to term 190
I1014 19:26:16.872483  5786 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 190 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:16.872570  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:16.872651  5786 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 190 FOLLOWER]: Advancing to term 191
I1014 19:26:16.874195  5786 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.874642  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 191 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.874840  5786 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 191 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.875424 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 191 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:16.875468 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 191 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.875635 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 190 FOLLOWER]: Advancing to term 191
I1014 19:26:16.875653 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 190 FOLLOWER]: Advancing to term 191
I1014 19:26:16.875720 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:16.875713 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (vote granted)
I1014 19:26:16.877130 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 191.
I1014 19:26:16.877432 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 191 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.877496 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 191 election: Election decided. Result: candidate won.
I1014 19:26:16.877705  5786 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:16.877763  5786 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 FOLLOWER]: Leader election won for term 191
I1014 19:26:16.877825 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 191 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 191.
I1014 19:26:16.877883  5786 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:16.878079  5786 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 189.120, Last appended by leader: 120, Current term: 191, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.878626 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 191 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:16.878921  6590 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 191 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.879225  6590 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.954167  6365 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:16.955129  6365 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:16.956063  6365 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 187 vs 191: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 191 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.956264  6365 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 191 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:16.956426  6365 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 191 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.976713 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 189 index: 120. Preceding OpId from leader: term: 191 index: 121. (index mismatch)
I1014 19:26:16.977242  6590 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I1014 19:26:16.978549  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 191 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.978786  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.980299  6585 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 191 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.980567  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 191 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:16.980641  6585 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:16.980676  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:16.980747  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 191 FOLLOWER]: Advancing to term 192
I1014 19:26:16.981321  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 191 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:16.981547  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:16.982278  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.982589 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 191. Current term is 192. Ops: []
I1014 19:26:16.982784  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 192 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:16.983022  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 192 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:16.983311  5644 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I1014 19:26:16.983644 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 192 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:16.983600  6585 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 LEADER]: Stepping down as leader of term 191
I1014 19:26:16.983803 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 192 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.983783  6585 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:16.983714 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 192 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:16.984004  6585 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 191.121, Last appended by leader: 121, Current term: 191, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:16.984207  6585 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:16.984211 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 192 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 192 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:16.984274  6585 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:16.984396  6585 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 191 FOLLOWER]: Advancing to term 192
I1014 19:26:16.985972 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 192 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 192 because replica has last-logged OpId of term: 191 index: 121, which is greater than that of the candidate, which has last-logged OpId of term: 189 index: 120.
I1014 19:26:16.986353 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 192 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 192 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 192 because replica has last-logged OpId of term: 191 index: 121, which is greater than that of the candidate, which has last-logged OpId of term: 189 index: 120.
I1014 19:26:16.986428 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 192 election: Election decided. Result: candidate lost.
I1014 19:26:16.986685  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.202s (election complete)
I1014 19:26:16.986757  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Leader election lost for term 192. Reason: could not achieve majority
I1014 19:26:17.082012  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:17.082134  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:17.082218  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 191 FOLLOWER]: Advancing to term 192
I1014 19:26:17.084630  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.085180  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 192 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.085422  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 192 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.085953 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 192 candidate_status { last_received { term: 191 index: 121 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.086169 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 192: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:17.086521 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 192 candidate_status { last_received { term: 191 index: 121 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.086714 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:17.088932 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 192 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 192.
I1014 19:26:17.089314 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 192 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.089381 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 192 election: Election decided. Result: candidate won.
I1014 19:26:17.089656  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:17.089741 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 192 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 192: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:17.089828  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 FOLLOWER]: Leader election won for term 192
I1014 19:26:17.096163  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.096405  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 191.121, Last appended by leader: 121, Current term: 192, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.103561  6775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 192 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.103794  6775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.189474  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:17.189560  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.173s (starting election)
I1014 19:26:17.189640  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 192 FOLLOWER]: Advancing to term 193
I1014 19:26:17.191121  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 193 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.191562  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 193 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.191730  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 193 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.192358 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 193 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.192531 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 192 FOLLOWER]: Advancing to term 193
I1014 19:26:17.192499 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 193 candidate_status { last_received { term: 189 index: 120 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.192693 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 193 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.193104 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 193 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 193 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.193799 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 193 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 192. Current term is 193. Ops: []
I1014 19:26:17.194345  6775 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I1014 19:26:17.194488 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 193 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 193 because replica has last-logged OpId of term: 191 index: 121, which is greater than that of the candidate, which has last-logged OpId of term: 189 index: 120.
I1014 19:26:17.194628  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 LEADER]: Stepping down as leader of term 192
I1014 19:26:17.194682  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.194793 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 193 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 193 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 193 because replica has last-logged OpId of term: 191 index: 121, which is greater than that of the candidate, which has last-logged OpId of term: 189 index: 120.
I1014 19:26:17.194859 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 193 election: Election decided. Result: candidate lost.
I1014 19:26:17.194878  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 192.122, Last appended by leader: 122, Current term: 192, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.195075  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:17.195071  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.251s (election complete)
I1014 19:26:17.195130  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:17.195163  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 193 FOLLOWER]: Leader election lost for term 193. Reason: could not achieve majority
I1014 19:26:17.195262  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 192 FOLLOWER]: Advancing to term 193
I1014 19:26:17.215973  6585 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 193 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:17.216058  6585 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:17.216138  6585 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 193 FOLLOWER]: Advancing to term 194
I1014 19:26:17.217641  6585 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.218070  6585 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 194 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.218232  6585 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 194 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.218855 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 194 candidate_status { last_received { term: 191 index: 121 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.219023 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 193 FOLLOWER]: Advancing to term 194
I1014 19:26:17.218966 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 194 candidate_status { last_received { term: 191 index: 121 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.219105 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.240s (vote granted)
I1014 19:26:17.219122 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 193 FOLLOWER]: Advancing to term 194
I1014 19:26:17.221194 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 194 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 194.
I1014 19:26:17.221344 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 194 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 194 because replica has last-logged OpId of term: 192 index: 122, which is greater than that of the candidate, which has last-logged OpId of term: 191 index: 121.
I1014 19:26:17.221683 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 194 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.221784 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 194 election: Election decided. Result: candidate won.
I1014 19:26:17.221904 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 194 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 194 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 194 because replica has last-logged OpId of term: 192 index: 122, which is greater than that of the candidate, which has last-logged OpId of term: 191 index: 121.
I1014 19:26:17.222107  6585 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:17.222180  6585 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 FOLLOWER]: Leader election won for term 194
I1014 19:26:17.222332  6585 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:17.222525  6585 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 191.121, Last appended by leader: 121, Current term: 194, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.223402  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 194 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.223680  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.292078  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 194 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:17.292182  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:17.292256  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 194 FOLLOWER]: Advancing to term 195
I1014 19:26:17.294452  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.295013  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 195 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.295411  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 195 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.295722 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 195 candidate_status { last_received { term: 192 index: 122 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.295907 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 194 FOLLOWER]: Advancing to term 195
I1014 19:26:17.295977 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.157s (vote granted)
I1014 19:26:17.297497 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 195 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 195.
I1014 19:26:17.297796 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 195 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.297783 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 195 candidate_status { last_received { term: 192 index: 122 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.297861 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 195 election: Election decided. Result: candidate won.
I1014 19:26:17.297940 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 195 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.298100  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:17.298239  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 FOLLOWER]: Leader election won for term 195
I1014 19:26:17.298287 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 195 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 195 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.298494  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.298769  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 192.122, Last appended by leader: 122, Current term: 195, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.299754  6775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.300014  6775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.330832 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 194. Current term is 195. Ops: []
I1014 19:26:17.331424  5644 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I1014 19:26:17.331626  5644 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 195.123 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 121
I1014 19:26:17.331732  5644 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 LEADER]: Stepping down as leader of term 194
I1014 19:26:17.331789  5644 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:17.332013  5644 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 194.122, Last appended by leader: 122, Current term: 194, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.332175  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:17.332252  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:17.332363  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 194 FOLLOWER]: Advancing to term 195
I1014 19:26:17.409592 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 195 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 189 index: 120. Preceding OpId from leader: term: 195 index: 123. (index mismatch)
I1014 19:26:17.410179  6110 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 120, Time since last communication: 0.000s
I1014 19:26:17.413070  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.413414  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.413183  6908 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.413818  6908 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.414398  6775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.414662  6775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.414963  6908 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.415280  6908 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.415314 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 194 index: 122. Preceding OpId from leader: term: 195 index: 123. (index mismatch)
I1014 19:26:17.415769  6908 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.415904  6775 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I1014 19:26:17.416064  6908 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.416131  6775 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 194.122 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 121
I1014 19:26:17.416770 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 121
I1014 19:26:17.416818 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 194.122
I1014 19:26:17.416879 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:17.417847  5786 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.418049  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.418048  6585 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.418507  5786 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.418602  6585 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.418510  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.511199  6775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 195 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.511636  6775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.515252  6908 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 195 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:17.515350  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:17.515426  6908 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 195 FOLLOWER]: Advancing to term 196
I1014 19:26:17.517606  6908 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 196 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.518077  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 196 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.518276  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 196 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.518741 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 196 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.518916 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 196 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.518887 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 196 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.519057 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 196 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.519249 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 196 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 196 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.519759 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 196 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 196 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.519834 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 196 election: Election decided. Result: candidate lost.
I1014 19:26:17.520287  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:17.520406  6908 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 196 FOLLOWER]: Leader election lost for term 196. Reason: could not achieve majority
I1014 19:26:17.520555 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 196 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 195. Current term is 196. Ops: []
I1014 19:26:17.521003  6775 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 195.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I1014 19:26:17.521224  6775 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Stepping down as leader of term 195
I1014 19:26:17.521275  6775 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.521536  6775 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 195.123, Last appended by leader: 123, Current term: 195, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.521690  6775 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:17.521756  6775 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:17.521879  6775 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 195 FOLLOWER]: Advancing to term 196
I1014 19:26:17.605069  6775 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 196 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:17.605159  6775 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:17.605245  6775 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 196 FOLLOWER]: Advancing to term 197
I1014 19:26:17.606853  6775 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.607272  6775 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 197 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.607488  6775 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 197 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.608052 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 197 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.608090 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 197 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.608192 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 196 FOLLOWER]: Advancing to term 197
I1014 19:26:17.608252 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 197 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.608255 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.216s (vote granted)
I1014 19:26:17.608593 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 197 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 197 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.609732 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 197.
I1014 19:26:17.610074 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 197 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.610139 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 197 election: Election decided. Result: candidate won.
I1014 19:26:17.610352  6775 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:17.610469  6775 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 FOLLOWER]: Leader election won for term 197
I1014 19:26:17.610584  6775 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.610934  6775 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 195.123, Last appended by leader: 123, Current term: 197, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.612360  6913 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.612596  6913 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.612989  5644 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:17.613080  5644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:17.613153  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 195 FOLLOWER]: Advancing to term 196
I1014 19:26:17.614573  5644 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 196 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.614940  5644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 196 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.615129  5644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 196 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.615622 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 196 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.615805 28639 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 196. Current term is 197.
I1014 19:26:17.615720 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 196 candidate_status { last_received { term: 195 index: 123 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.615906 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 196 because replica is either leader or believes a valid leader to be alive.
W1014 19:26:17.616253 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 196 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 196. Current term is 197.
I1014 19:26:17.616333 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 196 election: Cancelling election due to peer responding with higher term
W1014 19:26:17.616493 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 196 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 196 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.616647  5644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:17.616716  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 196 FOLLOWER]: Advancing to term 197
I1014 19:26:17.618001  5644 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Leader election lost for term 196. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 196. Current term is 197.
I1014 19:26:17.720002 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 195 index: 123. Preceding OpId from leader: term: 197 index: 124. (index mismatch)
I1014 19:26:17.720620  6913 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I1014 19:26:17.722028  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.722249  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.722892  6775 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.723100  6775 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.723742 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 195 index: 123. Preceding OpId from leader: term: 197 index: 124. (index mismatch)
I1014 19:26:17.723758  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.723958  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.724290  6110 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I1014 19:26:17.725682  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.726091  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.725908  6908 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.726215  6908 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:17.825706  6908 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:17.825795  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:17.825884  6908 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 197 FOLLOWER]: Advancing to term 198
I1014 19:26:17.827476  6908 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.827937  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 198 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.828135  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 198 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.828706 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 198 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.828876 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 198 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.828794 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 198 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.828974 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 198 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.829221 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 198 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 198 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.829399 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 198 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 198 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.829486 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 198 election: Election decided. Result: candidate lost.
I1014 19:26:17.829735  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:17.829808  6908 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Leader election lost for term 198. Reason: could not achieve majority
I1014 19:26:17.845079  6913 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 197 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.845553  6913 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.845691 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 197. Current term is 198. Ops: []
I1014 19:26:17.846249  6913 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 197.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I1014 19:26:17.846515  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Stepping down as leader of term 197
I1014 19:26:17.846573  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:17.846830  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 197.124, Last appended by leader: 124, Current term: 197, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.847024  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:17.847080  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:17.847193  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 197 FOLLOWER]: Advancing to term 198
I1014 19:26:17.914983  5644 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:17.915083  5644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (starting election)
I1014 19:26:17.915166  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 197 FOLLOWER]: Advancing to term 198
I1014 19:26:17.916738  5644 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.917248  5644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 198 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:17.917426  5644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 198 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.918004 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 198 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:17.918105 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 198 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.918198 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 198: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:17.918273 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:17.919255 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 198 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 198: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:17.920428 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 198 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 198.
I1014 19:26:17.920733 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 198 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.920814 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 198 election: Election decided. Result: candidate won.
I1014 19:26:17.921028  5644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:17.921097  5644 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 FOLLOWER]: Leader election won for term 198
I1014 19:26:17.921478  5644 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:17.921703  5644 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 197.124, Last appended by leader: 124, Current term: 198, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.922541  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 198 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.922806  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.937185  6908 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:17.937276  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.198s (starting election)
I1014 19:26:17.937367  6908 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 198 FOLLOWER]: Advancing to term 199
I1014 19:26:17.938951  6908 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.939397  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 199 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:17.939585  6908 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 199 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.940244 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 199 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:17.940325 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 199 candidate_status { last_received { term: 197 index: 124 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:17.940428 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 199 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.940518 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 198 FOLLOWER]: Advancing to term 199
I1014 19:26:17.940608 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:17.940783 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 199 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 199 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:17.942760 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 199 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 199.
I1014 19:26:17.943075 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 199 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:17.943137 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 199 election: Election decided. Result: candidate won.
I1014 19:26:17.943361  6908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.198s (election complete)
I1014 19:26:17.943457  6908 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 FOLLOWER]: Leader election won for term 199
I1014 19:26:17.943626  6908 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:17.943833  6908 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 197.124, Last appended by leader: 124, Current term: 199, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:17.945024  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:17.945214  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:17.945775  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
W1014 19:26:17.946156  6109 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.448s	user 0.002s	sys 0.000s
I1014 19:26:17.946223  6109 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:17.947219  6109 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:17.947777  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:17.948287  6109 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 188 vs 199: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.948532  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.948730  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.948916  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.949103  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.949314  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.949539  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:17.949735  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 199 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.021029 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 199 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 198. Current term is 199. Ops: []
I1014 19:26:18.021613  6937 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I1014 19:26:18.021889  5644 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 LEADER]: Stepping down as leader of term 198
I1014 19:26:18.021948  5644 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:18.022176  5644 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 198.125, Last appended by leader: 125, Current term: 198, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.022333  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:18.022419  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:18.022541  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 198 FOLLOWER]: Advancing to term 199
I1014 19:26:18.056486 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 199 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 197 index: 124. Preceding OpId from leader: term: 199 index: 125. (index mismatch)
I1014 19:26:18.057395  5596 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.001s
I1014 19:26:18.058655  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.058903  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.060008  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.060174  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.060464 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 198 index: 125. Preceding OpId from leader: term: 199 index: 125. (term mismatch)
I1014 19:26:18.060539 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 124
I1014 19:26:18.060577 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 198.125
I1014 19:26:18.060819  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.061038  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.061053 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:18.061617  5596 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I1014 19:26:18.062955  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.062953  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.063189  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.063343  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.153694  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 199 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.153977  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.161357  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 199 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:18.161525  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:18.161615  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 199 FOLLOWER]: Advancing to term 200
I1014 19:26:18.164662  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.165040 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 199. Current term is 200. Ops: []
I1014 19:26:18.165257  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 200 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.165505  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 200 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.166002 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 200 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.166221 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 200 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.166199 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 200 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.166409 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 200 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.166678  5596 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 199.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I1014 19:26:18.166880 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 200 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 200 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.166975  5596 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 LEADER]: Stepping down as leader of term 199
I1014 19:26:18.167044  5596 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:18.167078 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 200 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 200 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.167155 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 200 election: Election decided. Result: candidate lost.
I1014 19:26:18.167251  5596 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 199.125, Last appended by leader: 125, Current term: 199, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.167394  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:18.167503  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:18.167470  6110 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Leader election lost for term 200. Reason: could not achieve majority
I1014 19:26:18.167582  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:18.167695  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 199 FOLLOWER]: Advancing to term 200
I1014 19:26:18.255197  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:18.255307  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:18.255380  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 199 FOLLOWER]: Advancing to term 200
I1014 19:26:18.256920  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.257335  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 200 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.257526  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 200 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.258088 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 200 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.258137 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 200 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.258283 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:18.258371 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 200: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:18.258800 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 200 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 200: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:18.259843 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 200 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 200.
I1014 19:26:18.260167 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 200 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.260244 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 200 election: Election decided. Result: candidate won.
I1014 19:26:18.260500  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:18.260591  6937 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 FOLLOWER]: Leader election won for term 200
I1014 19:26:18.260732  6937 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:18.260952  6937 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 199.125, Last appended by leader: 125, Current term: 200, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.262133  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 200 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.262384  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.292162  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:18.292271  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.200s (starting election)
I1014 19:26:18.292351  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 200 FOLLOWER]: Advancing to term 201
I1014 19:26:18.294487  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.295069  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 201 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.295292  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 201 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.295946 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 201 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.296111 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 200 FOLLOWER]: Advancing to term 201
I1014 19:26:18.296208 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (vote granted)
I1014 19:26:18.296211 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 201 candidate_status { last_received { term: 199 index: 125 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.296422 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 201 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.296892 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 201 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 201 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.298288 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 201 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 201.
I1014 19:26:18.299016 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 201 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.299091 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 201 election: Election decided. Result: candidate won.
I1014 19:26:18.299337  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:18.299432  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 FOLLOWER]: Leader election won for term 201
I1014 19:26:18.299597  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:18.299824  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 199.125, Last appended by leader: 125, Current term: 201, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.300768  6913 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.300998  6913 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.301225  6109 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:18.302836  6109 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:18.356730 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 200. Current term is 201. Ops: []
I1014 19:26:18.357316  5644 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I1014 19:26:18.357564  5644 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 201.126 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 125
I1014 19:26:18.357686  5644 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 LEADER]: Stepping down as leader of term 200
I1014 19:26:18.357745  5644 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:18.358021  5644 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 200.126, Last appended by leader: 126, Current term: 200, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.358229  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:18.358301  5644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:18.358464  5644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 200 FOLLOWER]: Advancing to term 201
I1014 19:26:18.402344 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 201 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 199 index: 125. Preceding OpId from leader: term: 201 index: 126. (index mismatch)
I1014 19:26:18.402889  6913 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I1014 19:26:18.404115  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.404381  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.405407  6913 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.405833  6913 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.406594 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 201 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 200 index: 126. Preceding OpId from leader: term: 201 index: 126. (term mismatch)
I1014 19:26:18.406671 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 125
I1014 19:26:18.406710 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 200.126
I1014 19:26:18.406776 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 201 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:18.406838  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.407059  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.407471  6913 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I1014 19:26:18.408725  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.408996  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.409216  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.409389  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.507215  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 201 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:18.507324  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:18.507395  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 201 FOLLOWER]: Advancing to term 202
I1014 19:26:18.508669  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 201 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:18.508761  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:18.508826  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 201 FOLLOWER]: Advancing to term 202
I1014 19:26:18.509330  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.509831  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 202 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.510000  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 202 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.510186  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 201 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.510474  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.510938 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 202 candidate_status { last_received { term: 201 index: 126 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.510973 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 202 candidate_status { last_received { term: 201 index: 126 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.510926  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.511126 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 202 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.511334 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 201. Current term is 202. Ops: []
I1014 19:26:18.511517  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 202 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.511569 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 202 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 202 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.511723  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 202 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.511775 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 202 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:18.512301 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 202 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 202 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:18.512279  6110 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 201.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I1014 19:26:18.512420 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 202 election: Election decided. Result: candidate lost.
I1014 19:26:18.512507  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Stepping down as leader of term 201
I1014 19:26:18.512557  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:18.512562 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 202 candidate_status { last_received { term: 201 index: 126 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.512821 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 202: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:18.512732  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 201.126, Last appended by leader: 126, Current term: 201, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.512974  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:18.512976 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 202 candidate_status { last_received { term: 201 index: 126 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.513108  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:18.513177 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 202 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 202: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:18.513242  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 201 FOLLOWER]: Advancing to term 202
I1014 19:26:18.513319  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:18.513392  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Leader election lost for term 202. Reason: could not achieve majority
I1014 19:26:18.515420 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (vote granted)
I1014 19:26:18.516799 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 202.
I1014 19:26:18.517154 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 202 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.517223 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 202 election: Election decided. Result: candidate won.
I1014 19:26:18.517448  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:18.517526  6937 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Leader election won for term 202
I1014 19:26:18.517664  6937 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:18.517848  6937 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 201.126, Last appended by leader: 126, Current term: 202, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.518668  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.519003  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:18.549737  5182 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.979s	user 0.002s	sys 0.000s
I1014 19:26:18.549796  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:18.550796  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:18.553642 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 201 index: 126. Preceding OpId from leader: term: 202 index: 128. (index mismatch)
I1014 19:26:18.553645 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 201 index: 126. Preceding OpId from leader: term: 202 index: 128. (index mismatch)
I1014 19:26:18.554209  5644 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I1014 19:26:18.554467  6937 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I1014 19:26:18.556120  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.556479  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.556318  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.556547  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.558250  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.558444  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.558629  5644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.558861  5644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.560001  5182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:18.561830  5182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:18.562268  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.562511  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.563953  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 202 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.564308  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.745120  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:18.745250  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:18.745334  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 202 FOLLOWER]: Advancing to term 203
I1014 19:26:18.747781  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.748673  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 203 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.749004  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 203 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.749614 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 203 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.749822 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 203 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.749749 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 203 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.749987 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 203 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.750233 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 203 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 203 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.750452 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 203 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 203 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.750560 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 203 election: Election decided. Result: candidate lost.
I1014 19:26:18.751085  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:18.751377  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Leader election lost for term 203. Reason: could not achieve majority
I1014 19:26:18.766578 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 202. Current term is 203. Ops: []
I1014 19:26:18.767307  7078 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 202.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:18.767642  7078 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 LEADER]: Stepping down as leader of term 202
I1014 19:26:18.767709  7078 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:18.767899  7078 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 202.128, Last appended by leader: 128, Current term: 202, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.768123  7078 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:18.768177  7078 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:18.768337  7078 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 202 FOLLOWER]: Advancing to term 203
I1014 19:26:18.834404  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:18.834519  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:18.834604  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 202 FOLLOWER]: Advancing to term 203
I1014 19:26:18.836939  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.837509  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 203 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.837729  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 203 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.838297 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 203 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.838551 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 203: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:18.838603 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 203 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.838783 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:18.838954 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 203 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 203: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:18.840945 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 203 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 203.
I1014 19:26:18.841328 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 203 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.841392 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 203 election: Election decided. Result: candidate won.
I1014 19:26:18.841754  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:18.841852  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 FOLLOWER]: Leader election won for term 203
I1014 19:26:18.842074  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:18.842314  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 202.128, Last appended by leader: 128, Current term: 203, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.843617  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 203 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.843806  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.868575  7177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:18.868680  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (starting election)
I1014 19:26:18.868757  7177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 203 FOLLOWER]: Advancing to term 204
I1014 19:26:18.870928  7177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.871433  7177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 204 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.871659  7177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 204 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.872206 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 204 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:18.872289 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 204 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.872388 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 203 FOLLOWER]: Advancing to term 204
I1014 19:26:18.872503 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 204 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.872511 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (vote granted)
I1014 19:26:18.872902 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 204 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 204 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.874658 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 204 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 204.
I1014 19:26:18.875031 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 204 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:18.875114 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 204 election: Election decided. Result: candidate won.
I1014 19:26:18.875388  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.214s (election complete)
I1014 19:26:18.875505  7177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 FOLLOWER]: Leader election won for term 204
I1014 19:26:18.875638  7177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:18.875845  7177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 202.128, Last appended by leader: 128, Current term: 204, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.881348  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 204 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.881564  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.939926 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 204 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 203. Current term is 204. Ops: []
I1014 19:26:18.940598  6110 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:18.940857  6959 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 LEADER]: Stepping down as leader of term 203
I1014 19:26:18.940934  6959 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:18.941223  6959 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 203.129, Last appended by leader: 129, Current term: 203, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.941465  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:18.941574  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:18.941714  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 203 FOLLOWER]: Advancing to term 204
I1014 19:26:18.979977 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 203 index: 129. Preceding OpId from leader: term: 204 index: 129. (term mismatch)
I1014 19:26:18.980052 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 128
I1014 19:26:18.980088 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 203.129
I1014 19:26:18.980155 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:18.980741  5596 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:18.981915  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 204 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.982120  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.983609  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 204 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.983808  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:18.984145 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 204 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 202 index: 128. Preceding OpId from leader: term: 204 index: 129. (index mismatch)
I1014 19:26:18.984467  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 204 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:18.984544  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:18.984622  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 204 FOLLOWER]: Advancing to term 205
I1014 19:26:18.985049  7177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:18.985658  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 204 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:18.985878  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:18.986801  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.987139 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 204. Current term is 205. Ops: []
I1014 19:26:18.987319  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 205 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:18.987617  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 205 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:18.987852  7177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 202.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:18.988055  7177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 LEADER]: Stepping down as leader of term 204
I1014 19:26:18.988099  7177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:18.988267  7177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 204.129, Last appended by leader: 129, Current term: 204, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:18.988459  7177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:18.988539  7177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:18.988662  7177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 204 FOLLOWER]: Advancing to term 205
I1014 19:26:18.991319 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 205 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:18.991559 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 205 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 205 because replica has last-logged OpId of term: 204 index: 129, which is greater than that of the candidate, which has last-logged OpId of term: 202 index: 128.
I1014 19:26:18.992018 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 205 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 205 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 205 because replica has last-logged OpId of term: 204 index: 129, which is greater than that of the candidate, which has last-logged OpId of term: 202 index: 128.
I1014 19:26:18.992286 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 205 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:18.992502 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 205 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.992879 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 205 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 205 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:18.992955 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 205 election: Election decided. Result: candidate lost.
I1014 19:26:18.993192  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:18.993263  6937 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Leader election lost for term 205. Reason: could not achieve majority
I1014 19:26:19.085259  6959 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:19.085371  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:19.085467  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 204 FOLLOWER]: Advancing to term 205
I1014 19:26:19.087747  6959 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.088250  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 205 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.088415  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 205 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.089195 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 205 candidate_status { last_received { term: 204 index: 129 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.089380 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:19.091392 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 205 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 205.
I1014 19:26:19.091812 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 205 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.091883 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 205 election: Election decided. Result: candidate won.
I1014 19:26:19.092129  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:19.092245  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 FOLLOWER]: Leader election won for term 205
I1014 19:26:19.092355  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:19.092609  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 204.129, Last appended by leader: 129, Current term: 205, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.092705 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 205 candidate_status { last_received { term: 204 index: 129 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.092913 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 205: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:19.093515 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 205 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 205: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:19.094055  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 205 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.094360  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.110576  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:19.110682  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.192s (starting election)
I1014 19:26:19.110764  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 205 FOLLOWER]: Advancing to term 206
I1014 19:26:19.112999  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 206 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.113561  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 206 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.113780  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 206 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.114378 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 206 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.114575 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 205 FOLLOWER]: Advancing to term 206
I1014 19:26:19.116811 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 206 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 206 because replica has last-logged OpId of term: 204 index: 129, which is greater than that of the candidate, which has last-logged OpId of term: 202 index: 128.
I1014 19:26:19.117355 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 206 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 206 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 206 because replica has last-logged OpId of term: 204 index: 129, which is greater than that of the candidate, which has last-logged OpId of term: 202 index: 128.
I1014 19:26:19.117995 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 206 candidate_status { last_received { term: 202 index: 128 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.118202 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 206 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.118594 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 206 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 206 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.118674 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 206 election: Election decided. Result: candidate lost.
I1014 19:26:19.118916  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.218s (election complete)
I1014 19:26:19.118993  6937 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 206 FOLLOWER]: Leader election lost for term 206. Reason: could not achieve majority
I1014 19:26:19.181977 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 206 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 205. Current term is 206. Ops: []
I1014 19:26:19.182703  6959 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I1014 19:26:19.182965  6110 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 LEADER]: Stepping down as leader of term 205
I1014 19:26:19.183037  6110 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:19.183275  6110 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 205.130, Last appended by leader: 130, Current term: 205, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.183543  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:19.183614  6110 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:19.183758  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 205 FOLLOWER]: Advancing to term 206
I1014 19:26:19.227568  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 206 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:19.227656  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:19.227731  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 206 FOLLOWER]: Advancing to term 207
I1014 19:26:19.230041  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.230592  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 207 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.230799  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 207 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.231348 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 207 candidate_status { last_received { term: 204 index: 129 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.231570 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 206 FOLLOWER]: Advancing to term 207
I1014 19:26:19.231659 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.157s (vote granted)
I1014 19:26:19.231667 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 207 candidate_status { last_received { term: 204 index: 129 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.231855 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 206 FOLLOWER]: Advancing to term 207
I1014 19:26:19.233794 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 207 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 207.
I1014 19:26:19.234102 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 207 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 207 because replica has last-logged OpId of term: 205 index: 130, which is greater than that of the candidate, which has last-logged OpId of term: 204 index: 129.
I1014 19:26:19.234474 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 207 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.234599 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 207 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 207 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 207 because replica has last-logged OpId of term: 205 index: 130, which is greater than that of the candidate, which has last-logged OpId of term: 204 index: 129.
I1014 19:26:19.234683 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 207 election: Election decided. Result: candidate won.
I1014 19:26:19.234901  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:19.234972  7177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 FOLLOWER]: Leader election won for term 207
I1014 19:26:19.246809  7177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:19.247071  7177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 204.129, Last appended by leader: 129, Current term: 207, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.248651  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 207 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.248926  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.280539  6110 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 207 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:19.280678  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:19.280792  6110 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 207 FOLLOWER]: Advancing to term 208
I1014 19:26:19.283254  6110 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.284019  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 208 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.284349  6110 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 208 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.285043 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 208 candidate_status { last_received { term: 205 index: 130 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.285071 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 208 candidate_status { last_received { term: 205 index: 130 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.285203 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 207 FOLLOWER]: Advancing to term 208
I1014 19:26:19.285271 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 208 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.285284 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.238s (vote granted)
I1014 19:26:19.285804 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 208 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 208 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.287466 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 208.
I1014 19:26:19.287809 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 208 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.287878 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 208 election: Election decided. Result: candidate won.
I1014 19:26:19.288141  6110 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:19.288224  6110 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 FOLLOWER]: Leader election won for term 208
I1014 19:26:19.288398  6110 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:19.288635  6110 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 205.130, Last appended by leader: 130, Current term: 208, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.290112  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.290321  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.335570 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 207. Current term is 208. Ops: []
I1014 19:26:19.336236  7177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I1014 19:26:19.336498  7177 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 208.131 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 129
I1014 19:26:19.336611  7177 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 LEADER]: Stepping down as leader of term 207
I1014 19:26:19.336664  7177 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:19.336855  7177 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 207.130, Last appended by leader: 130, Current term: 207, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.337070  7177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:19.337136  7177 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:19.337294  7177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 207 FOLLOWER]: Advancing to term 208
I1014 19:26:19.365800 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 208 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 207 index: 130. Preceding OpId from leader: term: 208 index: 131. (index mismatch)
I1014 19:26:19.366291  6959 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s
I1014 19:26:19.366452  6959 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 207.130 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 129
I1014 19:26:19.367274 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 129
I1014 19:26:19.367326 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 207.130
I1014 19:26:19.367377 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 208 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:19.367977  7177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.368187  7177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.371278  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.371501  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.372413 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 202 index: 128. Preceding OpId from leader: term: 208 index: 131. (index mismatch)
I1014 19:26:19.372835  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.373061  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.373229  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.373419  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.373443  6110 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 128, Time since last communication: 0.000s
I1014 19:26:19.381222  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.382117  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.381258  7389 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.382653  7389 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.382356  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.382763  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.383016  7388 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.383388  7388 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.463765  6110 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 208 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.464010  6110 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.472673  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 208 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:19.472760  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:19.472828  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 208 FOLLOWER]: Advancing to term 209
I1014 19:26:19.475090  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.475630  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 209 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.476270  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 209 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.476402 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 209 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.476619 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 209 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.476905 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 209 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.477102 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 209 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.477632 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 209 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 209 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.477826 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 209 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 209 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.477900 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 209 election: Election decided. Result: candidate lost.
I1014 19:26:19.478150  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:19.478255  7177 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Leader election lost for term 209. Reason: could not achieve majority
I1014 19:26:19.493043 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 208. Current term is 209. Ops: []
I1014 19:26:19.497195  7418 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 208.131, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I1014 19:26:19.497567  6959 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Stepping down as leader of term 208
I1014 19:26:19.497642  6959 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:19.497900  6959 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 208.131, Last appended by leader: 131, Current term: 208, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.498117  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:19.498180  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:19.498356  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 208 FOLLOWER]: Advancing to term 209
I1014 19:26:19.564949  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:19.565052  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:19.565136  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 208 FOLLOWER]: Advancing to term 209
I1014 19:26:19.566694  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.567140  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 209 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.567342  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 209 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.567903 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 209 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.568029 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 209 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.568120 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 209: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:19.568208 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (vote granted)
I1014 19:26:19.569681 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 209 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 209.
I1014 19:26:19.570108 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 209 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.570178 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 209 election: Election decided. Result: candidate won.
I1014 19:26:19.570302 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 209 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 209: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:19.570540  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:19.570622  6937 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 FOLLOWER]: Leader election won for term 209
I1014 19:26:19.570765  6937 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:19.570964  6937 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 208.131, Last appended by leader: 131, Current term: 209, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.572051  7389 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 209 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.572186  7389 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.595722  7177 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:19.595832  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.160s (starting election)
I1014 19:26:19.595911  7177 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 209 FOLLOWER]: Advancing to term 210
I1014 19:26:19.597576  7177 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.598007  7177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 210 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.598198  7177 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 210 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.598956 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 210 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.599166 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 209 FOLLOWER]: Advancing to term 210
I1014 19:26:19.599274 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:19.601651 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 210 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 210.
I1014 19:26:19.602242 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 210 candidate_status { last_received { term: 208 index: 131 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.602457 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 210 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.603169 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 210 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 210 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.603296 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 210 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.603371 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 210 election: Election decided. Result: candidate won.
I1014 19:26:19.603646  7177 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.214s (election complete)
I1014 19:26:19.603719  7177 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 FOLLOWER]: Leader election won for term 210
I1014 19:26:19.603869  7177 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:19.604063  7177 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 208.131, Last appended by leader: 131, Current term: 210, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.605216  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.605402  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.649842 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 210 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 209. Current term is 210. Ops: []
I1014 19:26:19.650607  7389 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I1014 19:26:19.650851  7389 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 LEADER]: Stepping down as leader of term 209
I1014 19:26:19.650912  7389 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:19.651103  7389 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 209.132, Last appended by leader: 132, Current term: 209, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.651257  7389 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:19.651326  7389 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:19.651479  7389 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 209 FOLLOWER]: Advancing to term 210
W1014 19:26:19.658625  6939 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.711s	user 0.002s	sys 0.000s
I1014 19:26:19.658684  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:19.659636  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:19.662770 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 210 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 208 index: 131. Preceding OpId from leader: term: 210 index: 133. (index mismatch)
I1014 19:26:19.662971 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 209 index: 132. Preceding OpId from leader: term: 210 index: 133. (index mismatch)
I1014 19:26:19.663599  7177 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I1014 19:26:19.663743  7177 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 209.132 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 131
I1014 19:26:19.663892  5596 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I1014 19:26:19.664762 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 131
I1014 19:26:19.664811 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 209.132
I1014 19:26:19.664868 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:19.665709  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.665925  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.666195  7389 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.667986  7389 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.668447  7177 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.668356  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.668799  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.668803  7177 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.672405  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.672773  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.673337  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:19.675381  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:19.679518  6937 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 210 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.679872  6937 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.769260  6959 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 210 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:19.769371  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:26:19.769457  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 210 FOLLOWER]: Advancing to term 211
I1014 19:26:19.771905  6959 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.772589  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 211 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.773660  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 211 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.773754 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 211 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.773977 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 211 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.774451 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 211 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 211 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.774729 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 211 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.774924 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 211 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.775468 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 211 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 211 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.775588 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 211 election: Election decided. Result: candidate lost.
I1014 19:26:19.775869  7418 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:19.775964  7418 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Leader election lost for term 211. Reason: could not achieve majority
I1014 19:26:19.792858 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 210. Current term is 211. Ops: []
I1014 19:26:19.793388  7177 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 210.133, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I1014 19:26:19.793643  5596 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 LEADER]: Stepping down as leader of term 210
I1014 19:26:19.793736  5596 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:19.793916  5596 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 210.133, Last appended by leader: 133, Current term: 210, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.794134  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:19.794181  5596 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:19.794338  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 210 FOLLOWER]: Advancing to term 211
I1014 19:26:19.864281  6937 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:19.864375  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:19.864454  6937 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 210 FOLLOWER]: Advancing to term 211
I1014 19:26:19.866688  6937 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.867206  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 211 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.867396  6937 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 211 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:19.868127 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 211 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:19.868330 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 211: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:19.868515 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 211 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.868695 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (vote granted)
I1014 19:26:19.868818 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 211 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 211: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:19.870815 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 211 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 211.
I1014 19:26:19.871191 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 211 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.871273 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 211 election: Election decided. Result: candidate won.
I1014 19:26:19.871655  6937 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:19.871757  6937 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 FOLLOWER]: Leader election won for term 211
I1014 19:26:19.871974  6937 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:19.872205  6937 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 210.133, Last appended by leader: 133, Current term: 211, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.873694  7389 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 211 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.873908  7389 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.889880  7418 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:19.889994  7418 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.182s (starting election)
I1014 19:26:19.890074  7418 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 211 FOLLOWER]: Advancing to term 212
I1014 19:26:19.892355  7418 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.892916  7418 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 212 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.893144  7418 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 212 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:19.894099 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 212 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:19.894304 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 212 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.894366 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 212 candidate_status { last_received { term: 210 index: 133 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:19.894578 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 211 FOLLOWER]: Advancing to term 212
I1014 19:26:19.894676 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:19.894809 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 212 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 212 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:19.896741 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 212 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 212.
I1014 19:26:19.897169 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 212 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:19.897267 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 212 election: Election decided. Result: candidate won.
W1014 19:26:19.897790  5182 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.336s	user 0.002s	sys 0.000s
I1014 19:26:19.897856  5182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:19.898844  5182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:19.900207  7418 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:19.900295  7418 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 FOLLOWER]: Leader election won for term 212
I1014 19:26:19.900400  7418 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:19.900665  7418 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 210.133, Last appended by leader: 133, Current term: 212, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.901957  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.902197 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 211. Current term is 212. Ops: []
I1014 19:26:19.902434 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 212 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 211. Current term is 212. Ops: []
I1014 19:26:19.902221  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.903002  6937 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I1014 19:26:19.921423  6937 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 212.134 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 133
I1014 19:26:19.921640  7389 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I1014 19:26:19.921972  7513 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 LEADER]: Stepping down as leader of term 211
I1014 19:26:19.922206  7513 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:19.922718  7513 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 211.135, Last appended by leader: 135, Current term: 211, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:19.922966  7513 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:19.923058  7513 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:19.923247  7513 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 211 FOLLOWER]: Advancing to term 212
W1014 19:26:19.934684  6109 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.632s	user 0.002s	sys 0.000s
I1014 19:26:19.934756  6109 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:19.935724  6109 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:19.938349 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 212 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 210 index: 133. Preceding OpId from leader: term: 212 index: 135. (index mismatch)
I1014 19:26:19.938366 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 211 index: 135. Preceding OpId from leader: term: 212 index: 135. (term mismatch)
I1014 19:26:19.938419 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 134
I1014 19:26:19.938460 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 211.135
I1014 19:26:19.938962  5182 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 202 vs 212: Aborted: Transaction aborted by new leader
I1014 19:26:19.938956  6959 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
W1014 19:26:19.939291  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:19.939625  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.939558  6959 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I1014 19:26:19.939697  6959 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 211.134 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 133
W1014 19:26:19.939863  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:19.940230  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:19.940516  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.940562 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 133
I1014 19:26:19.940642 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 211.134
I1014 19:26:19.940701 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:19.942171  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.942409  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.942402  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.942631  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:19.943732  7418 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.943964  7418 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:19.944162  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.944478  6109 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
W1014 19:26:19.944520  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.944631  7513 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.944905  7513 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:19.945082  7513 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.945338  7513 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:19.946002  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:19.946245  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.946451  6109 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:19.946539  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.946501  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:19.946792  5182 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:19.946790  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:20.190220  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:20.191126  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:20.192236  6939 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 210 vs 212: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.192549  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.192782  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.192981  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.193188  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.193398  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.193611  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:20.193821  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 212 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.384781  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 212 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:20.384897  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:20.384976  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 212 FOLLOWER]: Advancing to term 213
I1014 19:26:20.387338  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.387959  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 213 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:20.388160  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 213 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.388792 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 213 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.388823 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 213 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:20.388980 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 213 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.389000 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 213 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.389453 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 213 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 213 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.389638 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 213 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 213 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.389742 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 213 election: Election decided. Result: candidate lost.
I1014 19:26:20.390048  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:20.390148  5596 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Leader election lost for term 213. Reason: could not achieve majority
I1014 19:26:20.399333 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 212. Current term is 213. Ops: []
I1014 19:26:20.399919  6110 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 212.135, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:20.400234  6959 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Stepping down as leader of term 212
I1014 19:26:20.400303  6959 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:20.400629  6959 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 212.135, Last appended by leader: 135, Current term: 212, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.400874  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:20.400985  6959 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:20.401141  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 212 FOLLOWER]: Advancing to term 213
I1014 19:26:20.494571  7705 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:20.494680  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:20.494765  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 212 FOLLOWER]: Advancing to term 213
I1014 19:26:20.497176  7705 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.497753  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 213 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.497983  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 213 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.498683 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 213 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:20.498900 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 213: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:20.498900 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 213 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.499088 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:20.499400 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 213 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 213: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:20.501128 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 213 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 213.
I1014 19:26:20.501480 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 213 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.501557 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 213 election: Election decided. Result: candidate won.
I1014 19:26:20.501966  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:20.502059  7705 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 FOLLOWER]: Leader election won for term 213
I1014 19:26:20.504828  7705 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:20.505089  7705 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 212.135, Last appended by leader: 135, Current term: 213, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.506301  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 213 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.506608  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.508136  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:20.510334  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:20.539903  5596 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:20.540012  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.217s (starting election)
I1014 19:26:20.540086  5596 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 213 FOLLOWER]: Advancing to term 214
I1014 19:26:20.542258  5596 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.542794  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 214 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:20.543421  5596 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 214 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.543520 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 214 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:20.543736 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 214 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.544082 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 214 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.544142 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 214 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 214 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.544240 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 213 FOLLOWER]: Advancing to term 214
I1014 19:26:20.544323 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:20.546149 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 214 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 214.
I1014 19:26:20.546483 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 214 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.546557 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 214 election: Election decided. Result: candidate won.
I1014 19:26:20.546767  5596 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.183s (election complete)
I1014 19:26:20.546847  5596 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 FOLLOWER]: Leader election won for term 214
I1014 19:26:20.547150  5596 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:20.547421  5596 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 212.135, Last appended by leader: 135, Current term: 214, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.549196  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 214 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.549757  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.549964  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:20.551524  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:20.616613 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 213. Current term is 214. Ops: []
I1014 19:26:20.617223  7712 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:20.617424  7712 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 214.136 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 135
I1014 19:26:20.617544  7712 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 LEADER]: Stepping down as leader of term 213
I1014 19:26:20.617589  7712 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:20.617777  7712 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 135, Committed index: 135, Last appended: 213.136, Last appended by leader: 136, Current term: 213, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.617987  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:20.618043  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:20.618132  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 213 FOLLOWER]: Advancing to term 214
I1014 19:26:20.619102 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 214 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 213. Current term is 214. Ops: []
I1014 19:26:20.641733 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 213 index: 136. Preceding OpId from leader: term: 214 index: 136. (term mismatch)
I1014 19:26:20.641805 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 135
I1014 19:26:20.641845 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 213.136
I1014 19:26:20.641898 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:20.642690  7757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:20.644045  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 214 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.644377  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:20.648319  6959 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 214 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:20.648407  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:20.648488  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 214 FOLLOWER]: Advancing to term 215
I1014 19:26:20.648437  5596 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 214 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.648699  5596 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.650858  6959 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.651388  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 215 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.651980 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 214. Current term is 215. Ops: []
I1014 19:26:20.652667 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 215 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:20.652869 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 215 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.653477  5596 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 136, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:20.653574 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 215 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 215 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.653692  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 215 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:20.654353 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 215 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:20.654500  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Stepping down as leader of term 214
I1014 19:26:20.654601  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:20.654880  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 214.136, Last appended by leader: 136, Current term: 214, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.655058  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:20.655107  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:20.655211  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 214 FOLLOWER]: Advancing to term 215
I1014 19:26:20.661042  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 214 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.661350 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 215 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:20.661350  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:20.661907 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 215 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 215 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:20.662036 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 215 election: Election decided. Result: candidate lost.
I1014 19:26:20.662420  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:20.662516  6959 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Leader election lost for term 215. Reason: could not achieve majority
I1014 19:26:20.745689  7705 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:20.745776  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:20.745848  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 214 FOLLOWER]: Advancing to term 215
I1014 19:26:20.747432  7705 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.747838  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 215 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.748010  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 215 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.748560 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 215 candidate_status { last_received { term: 214 index: 136 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:20.748708 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:20.748724 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 215 candidate_status { last_received { term: 214 index: 136 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.748960 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 215: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:20.749482 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 215 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 215: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:20.750324 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 215 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 215.
I1014 19:26:20.750737 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 215 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.750804 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 215 election: Election decided. Result: candidate won.
I1014 19:26:20.751031  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:20.751101  7705 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 FOLLOWER]: Leader election won for term 215
I1014 19:26:20.751204  7705 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:20.751474  7705 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 214.136, Last appended by leader: 136, Current term: 215, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.752678  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 215 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.752941  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.796872  6959 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:20.796978  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.188s (starting election)
I1014 19:26:20.797063  6959 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 215 FOLLOWER]: Advancing to term 216
I1014 19:26:20.798892  6959 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 216 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.799402  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 216 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.799633  6959 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 216 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:20.800292 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 216 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:20.800315 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 216 candidate_status { last_received { term: 212 index: 135 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:20.800479 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 215 FOLLOWER]: Advancing to term 216
I1014 19:26:20.800544 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 216 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.801314 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 216 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 216 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.802453 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 216 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 216 because replica has last-logged OpId of term: 214 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 212 index: 135.
I1014 19:26:20.802842 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 216 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 216 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 216 because replica has last-logged OpId of term: 214 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 212 index: 135.
I1014 19:26:20.802933 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 216 election: Election decided. Result: candidate lost.
I1014 19:26:20.803334  6959 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:20.803447  6959 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 216 FOLLOWER]: Leader election lost for term 216. Reason: could not achieve majority
I1014 19:26:20.844835 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 216 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 215. Current term is 216. Ops: []
I1014 19:26:20.845415  7712 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I1014 19:26:20.845723  7705 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 LEADER]: Stepping down as leader of term 215
I1014 19:26:20.845808  7705 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:20.846098  7705 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 215.137, Last appended by leader: 137, Current term: 215, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.846329  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:20.846431  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:20.846591  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 215 FOLLOWER]: Advancing to term 216
I1014 19:26:20.867506  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 216 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:20.867605  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:20.867667  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 216 FOLLOWER]: Advancing to term 217
I1014 19:26:20.869251  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.869791  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 217 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:20.869987  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 217 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.870612 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 217 candidate_status { last_received { term: 214 index: 136 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.870657 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 217 candidate_status { last_received { term: 214 index: 136 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:20.870790 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 216 FOLLOWER]: Advancing to term 217
I1014 19:26:20.870813 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 216 FOLLOWER]: Advancing to term 217
I1014 19:26:20.870877 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.288s (vote granted)
I1014 19:26:20.873108 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 217 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 217 because replica has last-logged OpId of term: 215 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 214 index: 136.
I1014 19:26:20.873126 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 217 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 217.
I1014 19:26:20.873558 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 217 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.873667 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 217 election: Election decided. Result: candidate won.
I1014 19:26:20.873800 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 217 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 217 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 217 because replica has last-logged OpId of term: 215 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 214 index: 136.
I1014 19:26:20.873914  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:20.874009  7757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 FOLLOWER]: Leader election won for term 217
I1014 19:26:20.874157  7757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:20.874359  7757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 214.136, Last appended by leader: 136, Current term: 217, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.875299  7835 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 217 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.875619  7835 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.937522  7705 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 217 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:20.937611  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:20.937690  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 217 FOLLOWER]: Advancing to term 218
I1014 19:26:20.939333  7705 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.939770  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 218 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:20.939954  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 218 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.940543 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 218 candidate_status { last_received { term: 215 index: 137 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:20.940750 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 218 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.940726 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 218 candidate_status { last_received { term: 215 index: 137 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:20.940896 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 217 FOLLOWER]: Advancing to term 218
I1014 19:26:20.940981 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.279s (vote granted)
I1014 19:26:20.942520 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 218 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 218.
I1014 19:26:20.942883 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 218 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:20.942983 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 218 election: Election decided. Result: candidate won.
I1014 19:26:20.943114 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 218 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 218 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:20.943280  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:20.943334  7705 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 FOLLOWER]: Leader election won for term 218
I1014 19:26:20.943469  7705 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:20.943691  7705 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 215.137, Last appended by leader: 137, Current term: 218, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.944700  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:20.944859  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:20.950825 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 218 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 217. Current term is 218. Ops: []
I1014 19:26:20.951326  7835 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 137, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:20.951617  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 LEADER]: Stepping down as leader of term 217
I1014 19:26:20.951674  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:20.951882  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 217.137, Last appended by leader: 137, Current term: 217, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:20.952044  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:20.952103  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:20.952211  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 217 FOLLOWER]: Advancing to term 218
I1014 19:26:21.010462  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:21.011440  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:21.017424 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 218 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 212 index: 135. Preceding OpId from leader: term: 218 index: 139. (index mismatch)
I1014 19:26:21.017938 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 218 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 217 index: 137. Preceding OpId from leader: term: 218 index: 139. (index mismatch)
I1014 19:26:21.018472  7705 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s
I1014 19:26:21.018635  7705 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 217.137 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 136
I1014 19:26:21.019528 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 136
I1014 19:26:21.019603 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 217.137
I1014 19:26:21.019709 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 218 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:21.022671  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.023227  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.023706  7712 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 135, Time since last communication: 0.000s
I1014 19:26:21.027457  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.027704  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.027940  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:21.030088  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:21.030841  7835 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.031635  7835 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.031635  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.031946  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.076086  6959 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.076373  6959 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.077293  7980 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.077492  7980 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.077755  7981 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.079109  7981 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.077828  7979 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.079315  7979 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.126056  7979 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 218 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.126183  7979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:21.126265  7979 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 218 FOLLOWER]: Advancing to term 219
I1014 19:26:21.127501  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 218 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.127637  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:21.127735  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 218 FOLLOWER]: Advancing to term 219
I1014 19:26:21.128654  7979 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.129154  7979 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 219 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.130008  7979 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 219 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.129926 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 219 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.130633 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 219 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.130831 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 219 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.130759  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 219 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.131188 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 219 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 219: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.131263  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 219 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.131646 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 219 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 219 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.131808 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 219 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 219 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 219: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.131876 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 219 election: Election decided. Result: candidate lost.
I1014 19:26:21.131927  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 219 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.132264 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 219 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.132438 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 219 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.132726 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 219 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.132841 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 219 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 219 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.132915 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 219: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:21.133191  7979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:21.133278 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 219 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 219: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:21.133296  7979 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Leader election lost for term 219. Reason: could not achieve majority
I1014 19:26:21.133438 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 219 election: Election decided. Result: candidate lost.
I1014 19:26:21.133752  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:21.133890  7757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 219 FOLLOWER]: Leader election lost for term 219. Reason: could not achieve majority
I1014 19:26:21.140995  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 218 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.141208  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.141340 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 218. Current term is 219. Ops: []
I1014 19:26:21.141886  7705 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 218.139, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I1014 19:26:21.142094  7705 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Stepping down as leader of term 218
I1014 19:26:21.142163  7705 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.142426  7705 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 218.139, Last appended by leader: 139, Current term: 218, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.142635  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:21.142699  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:21.142849  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 218 FOLLOWER]: Advancing to term 219
I1014 19:26:21.232477  7972 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 219 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:21.232590  7972 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:21.232678  7972 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 219 FOLLOWER]: Advancing to term 220
I1014 19:26:21.234292  7972 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.234771  7972 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 220 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.234928  7972 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 220 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.235502 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 220 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.235502 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 220 candidate_status { last_received { term: 218 index: 139 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.235664 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 219 FOLLOWER]: Advancing to term 220
I1014 19:26:21.235675 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 219 FOLLOWER]: Advancing to term 220
I1014 19:26:21.235754 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:21.235754 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:21.237246 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 220 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 220.
I1014 19:26:21.237640 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 220 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.237726 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 220 election: Election decided. Result: candidate won.
I1014 19:26:21.237901  7972 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:21.237972  7972 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 FOLLOWER]: Leader election won for term 220
I1014 19:26:21.238010 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 220 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 220.
I1014 19:26:21.238116  7972 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.238293  7972 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 218.139, Last appended by leader: 139, Current term: 220, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.238531 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 220 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.239338  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.239599  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.316540 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 220 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 218 index: 139. Preceding OpId from leader: term: 220 index: 140. (index mismatch)
I1014 19:26:21.317322  7705 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I1014 19:26:21.318742  7979 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.318967  7979 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.320713  7972 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.320986  7972 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.321293 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 220 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 218 index: 139. Preceding OpId from leader: term: 220 index: 140. (index mismatch)
I1014 19:26:21.321636  7979 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.321828  7979 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.322260  7972 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I1014 19:26:21.323771  7835 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.324007  7835 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.324136  7835 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.324331  7835 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.404793  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:21.405649  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:21.407414  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 220 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.407752  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.410034  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:21.412626  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:21.511430  7979 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 220 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.511535  7979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:21.511618  7979 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 220 FOLLOWER]: Advancing to term 221
I1014 19:26:21.512629  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 220 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.512728  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:21.512796  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 220 FOLLOWER]: Advancing to term 221
I1014 19:26:21.513295  7979 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 221 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.513743  7979 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 221 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.513944  7979 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 221 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.514806 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 221 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.514979 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 221 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.514873  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.515324  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 221 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.515339 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 221 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 221 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.515687  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 221 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.515693 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 221 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.515952 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 221: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.515913 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 221 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.516063 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 221 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.516304 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 221 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 221: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.516471 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 221 election: Election decided. Result: candidate lost.
I1014 19:26:21.516513 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 221 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.516770 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 221 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 221: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:21.516872  7979 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (election complete)
I1014 19:26:21.516973  7979 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 221 FOLLOWER]: Leader election lost for term 221. Reason: could not achieve majority
I1014 19:26:21.517168 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 221 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 221 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.517307 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 221 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 221 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 221: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:21.517407 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 221 election: Election decided. Result: candidate lost.
I1014 19:26:21.517665  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:21.517784  7757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Leader election lost for term 221. Reason: could not achieve majority
I1014 19:26:21.519572 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 220. Current term is 221. Ops: []
I1014 19:26:21.520031  7712 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 220.141, Next index: 142, Last known committed idx: 141, Time since last communication: 0.000s
I1014 19:26:21.520287  7705 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Stepping down as leader of term 220
I1014 19:26:21.520339  7705 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.520540  7705 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 141, Committed index: 141, Last appended: 220.141, Last appended by leader: 141, Current term: 220, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.520716  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:21.520767  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:21.520885  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 220 FOLLOWER]: Advancing to term 221
I1014 19:26:21.609273  7705 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 221 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:21.609375  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:21.609457  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 221 FOLLOWER]: Advancing to term 222
I1014 19:26:21.611068  7705 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.611511  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 222 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.611685  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 222 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.612340 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 222 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.612340 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 222 candidate_status { last_received { term: 220 index: 141 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.612536 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 221 FOLLOWER]: Advancing to term 222
I1014 19:26:21.612536 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 221 FOLLOWER]: Advancing to term 222
I1014 19:26:21.612623 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.167s (vote granted)
I1014 19:26:21.612623 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
I1014 19:26:21.614864 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 222 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 222.
I1014 19:26:21.614950 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 222.
I1014 19:26:21.615311 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 222 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.615406 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 222 election: Election decided. Result: candidate won.
I1014 19:26:21.615525 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 222 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.615694  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:21.615795  7705 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 FOLLOWER]: Leader election won for term 222
I1014 19:26:21.615942  7705 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.616140  7705 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 141, Committed index: 141, Last appended: 220.141, Last appended by leader: 141, Current term: 222, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.617111  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.617311  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.709740 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 222 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 220 index: 141. Preceding OpId from leader: term: 222 index: 142. (index mismatch)
I1014 19:26:21.710305  7712 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 142, Last known committed idx: 141, Time since last communication: 0.000s
I1014 19:26:21.711500  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.711800  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.712712  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.712908  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:21.713384  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.713613  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:21.715682  6109 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.769s	user 0.002s	sys 0.000s
I1014 19:26:21.715756  6109 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:21.716756  6109 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:21.717692 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 220 index: 141. Preceding OpId from leader: term: 222 index: 142. (index mismatch)
I1014 19:26:21.718821  7705 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 142, Last known committed idx: 141, Time since last communication: 0.000s
I1014 19:26:21.721699  7979 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.721956  7979 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:21.722033  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.722419  6109 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 212 vs 222: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 222 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.722460  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:21.722705  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.722972  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.723196  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.723477  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.723706  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.723959  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.724182  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.724406  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:21.724670  6109 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 222 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.813617  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 222 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.813724  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:21.813783  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 222 FOLLOWER]: Advancing to term 223
I1014 19:26:21.815315  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.815695  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 223 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.815877  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 223 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.816398 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 223 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.816637 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.816690 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 223 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.816890 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.817020 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 223 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.817356 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 223 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.817450 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 223 election: Election decided. Result: candidate lost.
I1014 19:26:21.817677  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:21.817776  7757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Leader election lost for term 223. Reason: could not achieve majority
I1014 19:26:21.820632  8273 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:21.820704  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:21.820766  8273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 222 FOLLOWER]: Advancing to term 223
I1014 19:26:21.823032  8273 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 223 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.823477  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 223 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.823662  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 223 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:21.823931 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 222. Current term is 223. Ops: []
I1014 19:26:21.824251 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 223 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:21.824458 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.824935  7705 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 222.142, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I1014 19:26:21.825063 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 223 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 223 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:21.825179  7705 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Stepping down as leader of term 222
I1014 19:26:21.825234  7705 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.825247 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 223 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.825500 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 223: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.825440  7705 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 222.142, Last appended by leader: 142, Current term: 222, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.825615  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:21.825680  7705 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:21.825827  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 222 FOLLOWER]: Advancing to term 223
I1014 19:26:21.825870 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 223 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 223: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:21.825949 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 223 election: Election decided. Result: candidate lost.
I1014 19:26:21.826407  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:21.826481  8273 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 223 FOLLOWER]: Leader election lost for term 223. Reason: could not achieve majority
I1014 19:26:21.920169  7705 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 223 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:21.920267  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:21.920339  7705 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 223 FOLLOWER]: Advancing to term 224
I1014 19:26:21.922536  7705 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.923079  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 224 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.923789  7705 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 224 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.923873 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 224 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:21.924048 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 223 FOLLOWER]: Advancing to term 224
I1014 19:26:21.924126 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.208s (vote granted)
I1014 19:26:21.924482 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 224 candidate_status { last_received { term: 222 index: 142 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:21.924656 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 223 FOLLOWER]: Advancing to term 224
I1014 19:26:21.924739 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:26:21.926199 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 224 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 224.
I1014 19:26:21.926635 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 224 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:21.926712 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 224 election: Election decided. Result: candidate won.
I1014 19:26:21.926812 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 224 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 224.
I1014 19:26:21.926939  7705 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:21.927081  7705 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 FOLLOWER]: Leader election won for term 224
I1014 19:26:21.927147 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 224 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:21.927263  7705 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:21.927564  7705 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 222.142, Last appended by leader: 142, Current term: 224, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:21.928864  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:21.929026  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.032313 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 224 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 222 index: 142. Preceding OpId from leader: term: 224 index: 143. (index mismatch)
I1014 19:26:22.032377 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 224 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 222 index: 142. Preceding OpId from leader: term: 224 index: 143. (index mismatch)
I1014 19:26:22.033004  7705 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I1014 19:26:22.033223  7712 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I1014 19:26:22.034601  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.034607  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.034823  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.034864  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.035595  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.035971  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.036599  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.036808  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.037175  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.037189  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.037324  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.037380  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:22.050642  6939 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.499s	user 0.002s	sys 0.000s
I1014 19:26:22.050706  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:22.051643  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:22.052922  6939 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 214 vs 224: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:22.053220  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:22.053537  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 224 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.137313  8273 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 224 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:22.137445  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:22.137550  8273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 224 FOLLOWER]: Advancing to term 225
I1014 19:26:22.139952  8273 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.140756  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 224 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:22.140856  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:22.140923  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 224 FOLLOWER]: Advancing to term 225
I1014 19:26:22.140913  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 225 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.141328  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 225 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.141811 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 225 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.142491 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 225 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.142735 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 225 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.143204 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 225 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 225 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.143384  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.143748 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 225: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:22.143980  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 225 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.144219  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 225 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.144243 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 225 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 225: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:22.144315 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 225 election: Election decided. Result: candidate lost.
I1014 19:26:22.144623  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:22.144706  8273 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Leader election lost for term 225. Reason: could not achieve majority
I1014 19:26:22.144651 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 225 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.144834 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 225 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.145102 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 225 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.145308 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 225: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:22.145720 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 225 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 225: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:22.145843 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 225 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 225 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.145928 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 225 election: Election decided. Result: candidate lost.
I1014 19:26:22.146169  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:22.146253  7757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Leader election lost for term 225. Reason: could not achieve majority
I1014 19:26:22.150733 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 224. Current term is 225. Ops: []
I1014 19:26:22.150846 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 224. Current term is 225. Ops: []
I1014 19:26:22.151319  8374 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 224.143, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I1014 19:26:22.151738  7712 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Stepping down as leader of term 224
I1014 19:26:22.151859  7712 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:22.151867  7705 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 224.143, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I1014 19:26:22.152249  7712 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 224.143, Last appended by leader: 143, Current term: 224, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.152482  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:22.152557  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:22.152688  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 224 FOLLOWER]: Advancing to term 225
I1014 19:26:22.254184  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 225 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:22.254284  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:22.254364  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 225 FOLLOWER]: Advancing to term 226
I1014 19:26:22.255893  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.256409  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 226 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.256621  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:22.256680  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 226 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.256714  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.208s (starting election)
I1014 19:26:22.256779  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 225 FOLLOWER]: Advancing to term 226
I1014 19:26:22.257374 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 226 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.257536 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 225 FOLLOWER]: Advancing to term 226
I1014 19:26:22.257625 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:22.259075 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 226.
I1014 19:26:22.258971  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.259395 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 226 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.259433  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 226 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.259459 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 226 election: Election decided. Result: candidate won.
I1014 19:26:22.259662  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 226 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.259711  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:22.259830  7712 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 FOLLOWER]: Leader election won for term 226
I1014 19:26:22.259997  7712 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:22.260116 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 226 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.260272 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 226 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.260253  7712 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 224.143, Last appended by leader: 143, Current term: 226, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.260445 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 226: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:22.260453 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 226 candidate_status { last_received { term: 224 index: 143 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.260632 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 226: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:22.260876 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 226 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 226: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:22.261461 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 226 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.261531 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 226 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 226: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:22.261831 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 226 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 226 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.261652  8374 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.261906 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 226 election: Election decided. Result: candidate lost.
I1014 19:26:22.261963  8374 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.262136  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:22.262238  7757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Leader election lost for term 226. Reason: could not achieve majority
I1014 19:26:22.348906 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 224 index: 143. Preceding OpId from leader: term: 226 index: 144. (index mismatch)
I1014 19:26:22.349483  8374 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I1014 19:26:22.350870  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.351086  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.352087 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 224 index: 143. Preceding OpId from leader: term: 226 index: 144. (index mismatch)
I1014 19:26:22.352015  7705 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.352250  7705 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.352648  7705 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I1014 19:26:22.353646  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.353807  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.354806  8480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.354950  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.355187  8480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.355242  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.454490  8273 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:22.454645  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:22.454733  8273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 226 FOLLOWER]: Advancing to term 227
I1014 19:26:22.457156  8273 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.457762  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 227 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.457965  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 227 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.458706 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 227 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.458714 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 227 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.458909 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 227 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.458921 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 227 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.459329 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 227 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 227 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.459542 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 227 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 227 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.459626 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 227 election: Election decided. Result: candidate lost.
I1014 19:26:22.459887  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:22.459997  8273 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Leader election lost for term 227. Reason: could not achieve majority
I1014 19:26:22.467200  8374 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 226 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.467459  8374 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.468228 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 226. Current term is 227. Ops: []
I1014 19:26:22.468785  8374 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 226.144, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I1014 19:26:22.469009  7712 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Stepping down as leader of term 226
I1014 19:26:22.469075  7712 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:22.469277  7712 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 226.144, Last appended by leader: 144, Current term: 226, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.469493  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:22.469578  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:22.469724  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 226 FOLLOWER]: Advancing to term 227
I1014 19:26:22.532726  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:22.532851  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:22.532930  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 226 FOLLOWER]: Advancing to term 227
I1014 19:26:22.535142  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.535681  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 227 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.535874  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 227 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.536818 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 227 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.537050 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 227: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:22.537358 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 227 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.537488 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 227 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 227: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:22.537544 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:22.539078 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 227 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 227.
I1014 19:26:22.539427 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 227 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.539510 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 227 election: Election decided. Result: candidate won.
I1014 19:26:22.539742  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:22.539834  7757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 FOLLOWER]: Leader election won for term 227
I1014 19:26:22.540256  7757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:22.540570  7757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 226.144, Last appended by leader: 144, Current term: 227, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.542019  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 227 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.542207  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.542476  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:22.544507  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:22.583030  8273 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:22.583129  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:22.583210  8273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 227 FOLLOWER]: Advancing to term 228
I1014 19:26:22.584888  8273 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.585294  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 228 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.585439  8273 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 228 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.586057 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 228 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.586145 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 228 candidate_status { last_received { term: 226 index: 144 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.586282 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 228 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.586320 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 227 FOLLOWER]: Advancing to term 228
I1014 19:26:22.586419 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:22.586776 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 228 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 228 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.588294 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 228 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 228.
I1014 19:26:22.588717 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 228 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.588784 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 228 election: Election decided. Result: candidate won.
I1014 19:26:22.589104  8273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:22.589184  8273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 FOLLOWER]: Leader election won for term 228
I1014 19:26:22.589314  8273 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:22.589563  8273 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 226.144, Last appended by leader: 144, Current term: 228, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.590536  8480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.590814  8480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.591398  8572 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:22.593072  8572 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:22.635144 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 228 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 227. Current term is 228. Ops: []
I1014 19:26:22.635776  8571 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I1014 19:26:22.636051  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 LEADER]: Stepping down as leader of term 227
I1014 19:26:22.636124  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:22.636387  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 227.145, Last appended by leader: 145, Current term: 227, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.636602  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:22.636654  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:22.636791  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 227 FOLLOWER]: Advancing to term 228
I1014 19:26:22.677794 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 227 index: 145. Preceding OpId from leader: term: 228 index: 145. (term mismatch)
I1014 19:26:22.677858 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 144
I1014 19:26:22.677891 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 227.145
I1014 19:26:22.677937 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:22.678505  8480 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I1014 19:26:22.679812  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.680066  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.681352  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.681627  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.682085 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 228 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 226 index: 144. Preceding OpId from leader: term: 228 index: 145. (index mismatch)
I1014 19:26:22.681974  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.682153  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.682616  8273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I1014 19:26:22.684118  8374 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.684418  8374 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.684340  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.684610  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:22.685454  8572 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:22.692873  8572 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:22.695379  8273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 228 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.695650  8273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.800770  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 228 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:22.800873  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:22.800945  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 228 FOLLOWER]: Advancing to term 229
I1014 19:26:22.802608  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.803102  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 229 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.803272  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 229 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.804010 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 229 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.804249 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 229 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.804276 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 229 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.804474 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 229 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.804663 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 229 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 229 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.804939 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 229 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 229 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.805022 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 229 election: Election decided. Result: candidate lost.
I1014 19:26:22.805352  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:22.805430  7712 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Leader election lost for term 229. Reason: could not achieve majority
I1014 19:26:22.808181 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 228. Current term is 229. Ops: []
I1014 19:26:22.808710  8480 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 228.146, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I1014 19:26:22.808993  8599 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 LEADER]: Stepping down as leader of term 228
I1014 19:26:22.809063  8599 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:22.809316  8599 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 228.146, Last appended by leader: 146, Current term: 228, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.809602  8599 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:22.809664  8599 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:22.809813  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 228 FOLLOWER]: Advancing to term 229
I1014 19:26:22.883306  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:22.883404  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:22.883510  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 228 FOLLOWER]: Advancing to term 229
I1014 19:26:22.885758  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.886301  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 229 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:22.886549  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 229 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.887029 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 229 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:22.887255 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 229: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:22.887302 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 229 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.887498 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:22.889717 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 229 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 229.
I1014 19:26:22.891386 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 229 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.891460 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 229 election: Election decided. Result: candidate won.
I1014 19:26:22.891819 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 229 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 229: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:22.892168  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:22.892256  7757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 FOLLOWER]: Leader election won for term 229
I1014 19:26:22.892360  7757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:22.892583  7757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 228.146, Last appended by leader: 146, Current term: 229, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.893998  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 229 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.894212  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.943744  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:22.943850  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.157s (starting election)
I1014 19:26:22.943922  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 229 FOLLOWER]: Advancing to term 230
I1014 19:26:22.945588  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.946017  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 230 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:22.946234  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 230 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.946802 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 230 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:22.946979 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 230 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.946907 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 230 candidate_status { last_received { term: 228 index: 146 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:22.947070 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 229 FOLLOWER]: Advancing to term 230
I1014 19:26:22.947155 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:22.947330 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 230 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 230 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:22.949388 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 230.
I1014 19:26:22.949772 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 230 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:22.949858 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 230 election: Election decided. Result: candidate won.
I1014 19:26:22.950114  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:22.950240  7712 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 FOLLOWER]: Leader election won for term 230
I1014 19:26:22.950387  7712 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:22.950685  7712 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 228.146, Last appended by leader: 146, Current term: 230, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.951568  8374 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:22.951750  8374 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:22.986110 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 229. Current term is 230. Ops: []
I1014 19:26:22.986711  7757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I1014 19:26:22.987004  7757 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 230.147 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 146
I1014 19:26:22.987166  8571 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 LEADER]: Stepping down as leader of term 229
I1014 19:26:22.987238  8571 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:22.987464  8571 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 229.147, Last appended by leader: 147, Current term: 229, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:22.987646  8571 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:22.987720  8571 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:22.987813  8571 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 229 FOLLOWER]: Advancing to term 230
W1014 19:26:23.039520  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.627s	user 0.002s	sys 0.000s
I1014 19:26:23.039580  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:23.040494  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:23.042697 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 230 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 229 index: 147. Preceding OpId from leader: term: 230 index: 148. (index mismatch)
I1014 19:26:23.042747 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 228 index: 146. Preceding OpId from leader: term: 230 index: 148. (index mismatch)
I1014 19:26:23.043213  8374 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I1014 19:26:23.043457  8374 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 229.147 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 146
I1014 19:26:23.043589  7712 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I1014 19:26:23.044340 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 146
I1014 19:26:23.044394 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 229.147
I1014 19:26:23.044474 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 230 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:23.045122  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.045353  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.045509  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.045755  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.047042  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.047238  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.047585  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:23.047487  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.047828  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.048974  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.049270  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.049540  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:23.049679  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 230 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.049883  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.149718  8571 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 230 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:23.149828  8571 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:23.149905  8571 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 230 FOLLOWER]: Advancing to term 231
I1014 19:26:23.151988  8571 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.152351 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 230. Current term is 231. Ops: []
I1014 19:26:23.152540  8571 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 231 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.153110  8571 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 231 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.153079  8743 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 230.148, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I1014 19:26:23.153429  7712 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 LEADER]: Stepping down as leader of term 230
I1014 19:26:23.153511  7712 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:23.153667 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 231 candidate_status { last_received { term: 230 index: 148 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.153755  7712 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 230.148, Last appended by leader: 148, Current term: 230, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.153940  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:23.154418  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:23.154593  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 230 FOLLOWER]: Advancing to term 231
I1014 19:26:23.154495 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 231 candidate_status { last_received { term: 230 index: 148 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:23.154719 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 231 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.155293 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 231 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 231 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.156419 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:23.157692 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 231.
I1014 19:26:23.158057 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 231 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.158130 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 231 election: Election decided. Result: candidate won.
I1014 19:26:23.158363  8571 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:23.158485  8571 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 FOLLOWER]: Leader election won for term 231
I1014 19:26:23.158623  8571 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:23.158825  8571 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 230.148, Last appended by leader: 148, Current term: 231, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.159837  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.160090  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.245362  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:23.245497  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:23.245586  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 230 FOLLOWER]: Advancing to term 231
I1014 19:26:23.247860  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 231 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.248402  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 231 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.248615  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 231 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.249336 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 231 candidate_status { last_received { term: 230 index: 148 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.249409 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 231 candidate_status { last_received { term: 230 index: 148 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.249580 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 231: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.249645 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 231 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.250069 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 231 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 231: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.250269 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 231 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 231 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.250349 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 231 election: Election decided. Result: candidate lost.
I1014 19:26:23.250577  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:23.250665  8599 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 231 FOLLOWER]: Leader election lost for term 231. Reason: could not achieve majority
I1014 19:26:23.275318 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 230 index: 148. Preceding OpId from leader: term: 231 index: 149. (index mismatch)
I1014 19:26:23.275854  7757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I1014 19:26:23.277055  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.277331  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.278288  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.278829  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.279004  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.279245  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.279691 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 231 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 230 index: 148. Preceding OpId from leader: term: 231 index: 149. (index mismatch)
I1014 19:26:23.280261  8571 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I1014 19:26:23.281720  8480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.281929  8480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.282163  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 231 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.282325  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.379884  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:23.379972  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:23.380084  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 231 FOLLOWER]: Advancing to term 232
I1014 19:26:23.381758  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 231 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:23.381711  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.381840  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:23.381911  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 231 FOLLOWER]: Advancing to term 232
I1014 19:26:23.382222  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 232 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.382396  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 232 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.383253 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 232 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:23.383975 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 232 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.384168 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 232 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.384253  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.384660 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 232 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 232 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.384752  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 232 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.384958 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 232: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:23.385406 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 232 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 232: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:23.385507 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 232 election: Election decided. Result: candidate lost.
I1014 19:26:23.385638  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 232 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.386467 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 232 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.386662 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 232 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.386749 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 232 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.386926 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 232: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:23.387126 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 232 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 232 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.387503 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 232 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 232: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:23.387611 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 232 election: Election decided. Result: candidate lost.
I1014 19:26:23.387657  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:23.387725  7712 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Leader election lost for term 232. Reason: could not achieve majority
I1014 19:26:23.387897  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:23.387974  8599 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Leader election lost for term 232. Reason: could not achieve majority
I1014 19:26:23.393062 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 231. Current term is 232. Ops: []
I1014 19:26:23.393548  8752 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 231.149, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I1014 19:26:23.393859  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Stepping down as leader of term 231
I1014 19:26:23.393929  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:23.394179  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 231.149, Last appended by leader: 149, Current term: 231, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.394388  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:23.394456  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:23.394631  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 231 FOLLOWER]: Advancing to term 232
I1014 19:26:23.498729  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 232 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:23.498842  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:23.498922  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 232 FOLLOWER]: Advancing to term 233
I1014 19:26:23.501260  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.501829  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 233 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.502022  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 233 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.502223  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:23.502313  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.155s (starting election)
I1014 19:26:23.502395  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 232 FOLLOWER]: Advancing to term 233
I1014 19:26:23.502732 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 233 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.502903 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 232 FOLLOWER]: Advancing to term 233
I1014 19:26:23.502981 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.169s (vote granted)
I1014 19:26:23.504592  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.505065  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 233 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.505132 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 233.
I1014 19:26:23.505239  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 233 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.505786 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 233 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.505820 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 233 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.505832 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 233 candidate_status { last_received { term: 231 index: 149 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:23.505956 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 233: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.505991 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 233: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.506011 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 233: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:23.506412 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 233 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 233: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.506464 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 233 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 233: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:23.506594 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 233 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 233: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:23.506609 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 233 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.506677 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 233 election: Election decided. Result: candidate lost.
I1014 19:26:23.506677 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 233 election: Election decided. Result: candidate won.
I1014 19:26:23.506929  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:23.506994  7757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 FOLLOWER]: Leader election won for term 233
I1014 19:26:23.507035  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.184s (election complete)
I1014 19:26:23.507117  8599 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Leader election lost for term 233. Reason: could not achieve majority
I1014 19:26:23.507530  7757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:23.507741  7757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 231.149, Last appended by leader: 149, Current term: 233, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.508780  8752 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.509068  8752 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.595549 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 231 index: 149. Preceding OpId from leader: term: 233 index: 150. (index mismatch)
I1014 19:26:23.596098  8752 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I1014 19:26:23.597543  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.597815  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.598630  8752 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.598832  8752 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.599239  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.599618  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.599802 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 231 index: 149. Preceding OpId from leader: term: 233 index: 150. (index mismatch)
I1014 19:26:23.600299  8752 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I1014 19:26:23.601449  8743 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.601729  8743 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.601888  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.602052  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:23.651566  6939 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.107s	user 0.002s	sys 0.000s
I1014 19:26:23.651648  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:23.652798  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:23.654810  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 233 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.655089  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.665071  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:23.666733  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:23.758648  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:23.758744  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:23.758824  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 233 FOLLOWER]: Advancing to term 234
I1014 19:26:23.761040  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.761591  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 234 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.761768  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 234 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.762037  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:23.762156  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:23.762315  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 233 FOLLOWER]: Advancing to term 234
I1014 19:26:23.762457 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 234 candidate_status { last_received { term: 233 index: 151 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.762691 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.763152 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 234 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.763118 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 234 candidate_status { last_received { term: 233 index: 151 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:23.764806  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.765331 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.765705  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 234 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:23.765820 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 234 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.765908 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 234 election: Election decided. Result: candidate lost.
I1014 19:26:23.766095  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 234 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.766301 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 234 candidate_status { last_received { term: 233 index: 151 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:23.766507 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.766667  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:23.766770 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 234 candidate_status { last_received { term: 233 index: 151 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.766867 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 234 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 234 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:23.766965  7712 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Leader election lost for term 234. Reason: could not achieve majority
I1014 19:26:23.767086 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 234: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:23.767475 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 234 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 234: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:23.767580 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 234 election: Election decided. Result: candidate lost.
I1014 19:26:23.767868  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:23.767941  8599 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Leader election lost for term 234. Reason: could not achieve majority
I1014 19:26:23.772585  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:23.773514  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:23.776072 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 233. Current term is 234. Ops: []
I1014 19:26:23.776121 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 233. Current term is 234. Ops: []
I1014 19:26:23.776775  8571 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 233.151, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I1014 19:26:23.777115  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Stepping down as leader of term 233
I1014 19:26:23.777228  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:23.777108  8761 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 233.151, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I1014 19:26:23.777554  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 233.152, Last appended by leader: 152, Current term: 233, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.777746  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:23.777810  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:23.777935  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 233 FOLLOWER]: Advancing to term 234
I1014 19:26:23.866657  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 234 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:23.866760  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:23.866839  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 234 FOLLOWER]: Advancing to term 235
I1014 19:26:23.868978  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.869541  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 235 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.870016  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 235 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.870728 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 235 candidate_status { last_received { term: 233 index: 152 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:23.870884 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 234 FOLLOWER]: Advancing to term 235
I1014 19:26:23.870963 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.196s (vote granted)
I1014 19:26:23.871290 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 235 candidate_status { last_received { term: 233 index: 152 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:23.871465 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 234 FOLLOWER]: Advancing to term 235
I1014 19:26:23.871569 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.186s (vote granted)
I1014 19:26:23.873085 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 235.
I1014 19:26:23.873669 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 235 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 235.
I1014 19:26:23.873742 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 235 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:23.873813 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 235 election: Election decided. Result: candidate won.
I1014 19:26:23.874054 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 235 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:23.874411  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:23.874514  7757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 FOLLOWER]: Leader election won for term 235
I1014 19:26:23.874843  7757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:23.875229  7757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 233.152, Last appended by leader: 152, Current term: 235, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:23.876606  8761 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.876883  8761 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.988298 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 233 index: 151. Preceding OpId from leader: term: 235 index: 153. (index mismatch)
I1014 19:26:23.989120  7757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 151, Time since last communication: 0.000s
I1014 19:26:23.989578 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 235 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 233 index: 151. Preceding OpId from leader: term: 235 index: 153. (index mismatch)
I1014 19:26:23.990052  7757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 151, Time since last communication: 0.000s
I1014 19:26:23.991510  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.991829  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.992779  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.993019  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.993127  6939 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:23.993585  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.993822  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.994113  8571 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.994488  8571 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:23.995478  6939 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:23.995512  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.995745  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:23.996635  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 235 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:23.996990  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.096194  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 235 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:24.096315  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:24.096395  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 235 FOLLOWER]: Advancing to term 236
I1014 19:26:24.098731  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.099278  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 236 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.099676  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 236 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:24.099941 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 236 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.100162 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 236 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.100282 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 236 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:24.100469 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 236 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.100654 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 236 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 236 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.100817 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 236 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 236 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.100879 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 236 election: Election decided. Result: candidate lost.
I1014 19:26:24.101090  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:24.101167  8599 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Leader election lost for term 236. Reason: could not achieve majority
I1014 19:26:24.119212 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 235. Current term is 236. Ops: []
I1014 19:26:24.119827  8761 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 235.153, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.120126  7757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 LEADER]: Stepping down as leader of term 235
I1014 19:26:24.120180  7757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:24.120419  7757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 235.153, Last appended by leader: 153, Current term: 235, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.120577  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:24.120640  7757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:24.120743  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 235 FOLLOWER]: Advancing to term 236
I1014 19:26:24.192175  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:24.192296  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:24.192385  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 235 FOLLOWER]: Advancing to term 236
I1014 19:26:24.194842  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.195502  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 236 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.195772  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 236 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.196322 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 236 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.196533 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (vote granted)
I1014 19:26:24.196660 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 236 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:24.196871 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 236: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:24.197515 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 236 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 236: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:24.198333 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 236 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 236.
I1014 19:26:24.198673 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 236 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.198742 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 236 election: Election decided. Result: candidate won.
I1014 19:26:24.199066  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:24.199179  7712 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 FOLLOWER]: Leader election won for term 236
I1014 19:26:24.199694  7712 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:24.200024  7712 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 235.153, Last appended by leader: 153, Current term: 236, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.201314  8853 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 236 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.201597  8853 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.204238  8599 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:24.204357  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.189s (starting election)
I1014 19:26:24.204454  8599 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 236 FOLLOWER]: Advancing to term 237
I1014 19:26:24.207033  8599 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.207795  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 237 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.208113  8599 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 237 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:24.208706 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 237 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.208863 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 236 FOLLOWER]: Advancing to term 237
I1014 19:26:24.208950 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (vote granted)
I1014 19:26:24.208869 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 237 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:24.209128 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 237 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.209583 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 237 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 237 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.210600 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 237 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 237.
I1014 19:26:24.210891 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 237 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.210958 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 237 election: Election decided. Result: candidate won.
I1014 19:26:24.211215  8599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.182s (election complete)
I1014 19:26:24.211298  8599 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 FOLLOWER]: Leader election won for term 237
I1014 19:26:24.211688  8599 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:24.211957  8599 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 235.153, Last appended by leader: 153, Current term: 237, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.213562  8854 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 237 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.213834  8854 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.214342  8855 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:24.215898  8855 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:24.304486 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 236. Current term is 237. Ops: []
I1014 19:26:24.305073  8853 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.305284  8853 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 237.154 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 153
I1014 19:26:24.305385  7712 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 LEADER]: Stepping down as leader of term 236
I1014 19:26:24.305485  7712 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:24.305763  7712 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 236.154, Last appended by leader: 154, Current term: 236, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.305958  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:24.306020  7712 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:24.306149  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 236 FOLLOWER]: Advancing to term 237
I1014 19:26:24.318778  7757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 237 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:24.318876  7757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:24.318949  7757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 237 FOLLOWER]: Advancing to term 238
I1014 19:26:24.321188  7757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 238 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.321756  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 238 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:24.322566 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 238 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:24.322731 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 237 FOLLOWER]: Advancing to term 238
I1014 19:26:24.322841  7757 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 238 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.323068 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 238 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 237. Current term is 238. Ops: []
I1014 19:26:24.323416 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 238 candidate_status { last_received { term: 235 index: 153 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:24.323693 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 238 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.324241 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 238 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 238 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.324633 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 238 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 238 because replica has last-logged OpId of term: 236 index: 154, which is greater than that of the candidate, which has last-logged OpId of term: 235 index: 153.
I1014 19:26:24.324689  8854 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.324926  8854 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Stepping down as leader of term 237
I1014 19:26:24.324988 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 238 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 237. Current term is 238. Ops: []
I1014 19:26:24.325001  8854 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:24.325052 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 238 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 238 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 238 because replica has last-logged OpId of term: 236 index: 154, which is greater than that of the candidate, which has last-logged OpId of term: 235 index: 153.
I1014 19:26:24.325161 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 238 election: Election decided. Result: candidate lost.
I1014 19:26:24.325213  8854 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 237.154, Last appended by leader: 154, Current term: 237, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.325392  8761 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:24.325440  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:24.325501  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:24.325515  8761 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 238 FOLLOWER]: Leader election lost for term 238. Reason: could not achieve majority
I1014 19:26:24.325628  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 237 FOLLOWER]: Advancing to term 238
I1014 19:26:24.404884  7712 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 238 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:24.404992  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:24.405088  7712 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 238 FOLLOWER]: Advancing to term 239
I1014 19:26:24.407318  7712 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.407892  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 239 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.408684 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 239 candidate_status { last_received { term: 236 index: 154 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.408855 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 238 FOLLOWER]: Advancing to term 239
I1014 19:26:24.408862  7712 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 239 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.408932 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.158s (vote granted)
I1014 19:26:24.409516 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 239 candidate_status { last_received { term: 236 index: 154 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:24.409674 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 238 FOLLOWER]: Advancing to term 239
I1014 19:26:24.411072 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 239 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 239.
I1014 19:26:24.411588 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 239 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.411654 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 239 election: Election decided. Result: candidate won.
I1014 19:26:24.411911 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 239 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 239 because replica has last-logged OpId of term: 237 index: 154, which is greater than that of the candidate, which has last-logged OpId of term: 236 index: 154.
I1014 19:26:24.411911  7712 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:24.411981  7712 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 FOLLOWER]: Leader election won for term 239
I1014 19:26:24.412219  7712 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:24.412307 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 239 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 239 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 239 because replica has last-logged OpId of term: 237 index: 154, which is greater than that of the candidate, which has last-logged OpId of term: 236 index: 154.
I1014 19:26:24.412539  7712 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 236.154, Last appended by leader: 154, Current term: 239, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.413607  8853 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 239 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.413851  8853 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.436354  8854 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 239 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:24.436480  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:24.436555  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 239 FOLLOWER]: Advancing to term 240
I1014 19:26:24.438086  8854 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.438637  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 240 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.438839  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 240 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:24.439541 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 240 candidate_status { last_received { term: 237 index: 154 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:24.439654 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 240 candidate_status { last_received { term: 237 index: 154 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.439748 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 240 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.439821 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 239 FOLLOWER]: Advancing to term 240
I1014 19:26:24.439903 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.219s (vote granted)
I1014 19:26:24.440289 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 240 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 240 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:24.441601 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 240 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 240.
I1014 19:26:24.442607 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 240 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.442679 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 240 election: Election decided. Result: candidate won.
I1014 19:26:24.442929  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:24.443011  8854 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 FOLLOWER]: Leader election won for term 240
I1014 19:26:24.443164  8854 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:24.443403  8854 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 237.154, Last appended by leader: 154, Current term: 240, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.444803  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.445026  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.505203 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 239. Current term is 240. Ops: []
I1014 19:26:24.505692  7712 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.505906  7712 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 240.155 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 153
I1014 19:26:24.506044  8853 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 LEADER]: Stepping down as leader of term 239
I1014 19:26:24.506098  8853 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:24.506319  8853 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 239.155, Last appended by leader: 155, Current term: 239, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.506479  8853 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:24.506542  8853 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:24.506688  8853 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 239 FOLLOWER]: Advancing to term 240
I1014 19:26:24.547051 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 240 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 239 index: 155. Preceding OpId from leader: term: 240 index: 155. (term mismatch)
I1014 19:26:24.547139 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 154
I1014 19:26:24.547196 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 239.155
I1014 19:26:24.547296 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 240 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:24.548023  8854 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.548190  8854 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 236.154 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 153
I1014 19:26:24.549113 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 153
I1014 19:26:24.549173 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 236.154
I1014 19:26:24.549255 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 240 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:24.550007  8853 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.550416  8853 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.551527  8599 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.551820  8599 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.552772 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 240 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 235 index: 153. Preceding OpId from leader: term: 240 index: 155. (index mismatch)
I1014 19:26:24.553310  8599 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 153, Time since last communication: 0.000s
I1014 19:26:24.558118  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.558089  8761 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.558411  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.558432  8761 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.558635  7757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.558876  7757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.560204  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.560483  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.560634  8853 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 240 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.560780  8853 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.652470  8853 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 240 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:24.652549  8853 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:24.652623  8853 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 240 FOLLOWER]: Advancing to term 241
I1014 19:26:24.654460  8853 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.654956  8853 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 241 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:24.655032 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 240. Current term is 241. Ops: []
I1014 19:26:24.655272  8853 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 241 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.655665  8922 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 240 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:24.655750  8922 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:24.655823  8922 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 240 FOLLOWER]: Advancing to term 241
I1014 19:26:24.655711  8919 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 240.155, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I1014 19:26:24.656040  8854 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 LEADER]: Stepping down as leader of term 240
I1014 19:26:24.656111  8854 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:24.656301 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 241 candidate_status { last_received { term: 240 index: 155 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:24.656350  8854 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 240.155, Last appended by leader: 155, Current term: 240, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.656621  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:24.656682  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:24.656709 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 241 candidate_status { last_received { term: 240 index: 155 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:24.656842  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 240 FOLLOWER]: Advancing to term 241
I1014 19:26:24.658033  8922 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.658339 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 241: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:24.658520  8922 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 241 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:24.658799  8922 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 241 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.658845 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 241 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 241: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:24.659016 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:24.659373 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 241 candidate_status { last_received { term: 240 index: 155 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:24.659373 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 241 candidate_status { last_received { term: 240 index: 155 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:24.659575 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 241: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:24.659875 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 241 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 241: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:24.661101 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 241.
I1014 19:26:24.661260 29010 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 241 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:24.661536 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 241 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:24.661623 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 241 election: Election decided. Result: candidate won.
I1014 19:26:24.661707 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 241 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Service unavailable: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 241 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:24.661782 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 241 election: Election decided. Result: candidate lost.
I1014 19:26:24.661969  8853 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (election complete)
I1014 19:26:24.661999  8922 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:24.662047  8853 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Leader election won for term 241
I1014 19:26:24.662073  8922 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Leader election lost for term 241. Reason: could not achieve majority
I1014 19:26:24.662173  8853 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:24.662456  8853 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 240.155, Last appended by leader: 155, Current term: 241, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:24.663484  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.663677  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.751215 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 240 index: 155. Preceding OpId from leader: term: 241 index: 156. (index mismatch)
I1014 19:26:24.751803  7712 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I1014 19:26:24.753186  8922 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.753556  8922 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.754436  8853 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.754690  8853 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:24.754868  8922 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.755177  8922 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.755789 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 240 index: 155. Preceding OpId from leader: term: 241 index: 156. (index mismatch)
I1014 19:26:24.756227  8853 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I1014 19:26:24.757572  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.757903  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.757979  8854 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.758273  8854 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:24.841920  7712 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 241 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:24.842159  7712 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:25.019707  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.970s	user 0.002s	sys 0.000s
I1014 19:26:25.019798  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:25.020879  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:25.032191  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:25.034147  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:25.129503  8854 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:25.129604  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:25.129671  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 241 FOLLOWER]: Advancing to term 242
I1014 19:26:25.131186  8854 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.131631  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 242 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.131837  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 242 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.132429 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 242 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:25.132433 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 242 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:25.132653 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 242 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.132659 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 242 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.133134 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 242 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 242 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.133309 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 242 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 242 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.133376 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 242 election: Election decided. Result: candidate lost.
I1014 19:26:25.133596  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:25.133669  8854 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Leader election lost for term 242. Reason: could not achieve majority
I1014 19:26:25.144279 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 241. Current term is 242. Ops: []
I1014 19:26:25.144948  8853 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 241.157, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I1014 19:26:25.145218  8969 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 LEADER]: Stepping down as leader of term 241
I1014 19:26:25.145308  8969 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:25.145582  8969 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 241.157, Last appended by leader: 157, Current term: 241, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.145828  8969 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:25.145895  8969 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:25.146034  8969 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 241 FOLLOWER]: Advancing to term 242
I1014 19:26:25.219926  8922 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:25.220022  8922 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:25.220122  8922 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 241 FOLLOWER]: Advancing to term 242
I1014 19:26:25.222259  8922 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.222792  8922 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 242 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.223645 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 242 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:25.223831 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:25.225927 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 242 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 242.
I1014 19:26:25.226138  8922 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 242 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:25.226675 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 242 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.226745 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 242 election: Election decided. Result: candidate won.
I1014 19:26:25.226799 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 242 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:25.227003 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 242: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:25.227028  8922 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:25.227095  8922 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 FOLLOWER]: Leader election won for term 242
I1014 19:26:25.227195  8922 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:25.227399  8922 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 241.157, Last appended by leader: 157, Current term: 242, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.227618 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 242 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 242: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:25.228447  8922 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 242 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.228718  8922 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.240742  8854 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:25.240852  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.192s (starting election)
I1014 19:26:25.240924  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 242 FOLLOWER]: Advancing to term 243
I1014 19:26:25.243137  8854 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.243646  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 243 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.243857  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 243 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.244433 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 243 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:25.244432 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 243 candidate_status { last_received { term: 241 index: 157 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:25.244637 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 242 FOLLOWER]: Advancing to term 243
I1014 19:26:25.244665 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 243 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.244720 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (vote granted)
I1014 19:26:25.245072 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 243 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 243 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.246850 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 243 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 243.
I1014 19:26:25.247567 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 243 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.247642 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 243 election: Election decided. Result: candidate won.
I1014 19:26:25.248555  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:25.248641  8854 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 FOLLOWER]: Leader election won for term 243
I1014 19:26:25.248786  8854 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:25.249008  8854 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 241.157, Last appended by leader: 157, Current term: 243, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.249922  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.250183  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:25.301368  6939 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.306s	user 0.001s	sys 0.000s
I1014 19:26:25.301430  6939 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:25.302350  6939 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:25.304692 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 242. Current term is 243. Ops: []
I1014 19:26:25.304689 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 243 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 242. Current term is 243. Ops: []
I1014 19:26:25.305356  9049 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I1014 19:26:25.312306  9049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 243.158 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 157
I1014 19:26:25.312639  8922 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I1014 19:26:25.312866  9049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 LEADER]: Stepping down as leader of term 242
I1014 19:26:25.312947  9049 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:25.313244  9049 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 242.159, Last appended by leader: 159, Current term: 242, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.313477  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:25.313591  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:25.313760  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 242 FOLLOWER]: Advancing to term 243
I1014 19:26:25.339700 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 243 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 241 index: 157. Preceding OpId from leader: term: 243 index: 158. (index mismatch)
I1014 19:26:25.340315  8919 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I1014 19:26:25.341621  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.341995  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.342955  8854 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.343235  8854 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.343684 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 242 index: 159. Preceding OpId from leader: term: 243 index: 158. (term mismatch)
I1014 19:26:25.343756 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 157
I1014 19:26:25.343794 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 242.158
I1014 19:26:25.343855 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:25.343917 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted WRITE_OP operation due to leader change: 242.159
I1014 19:26:25.344629  6939 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 235 vs 243: Aborted: Transaction aborted by new leader
W1014 19:26:25.344965  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.344839  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.345060  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:25.345201  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:25.345435  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:25.345758  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:25.346052  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:25.346251  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.346244  9067 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
W1014 19:26:25.346505  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:25.346716  6939 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 243 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.349130  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.349319  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.349467  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.349673  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.428637  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 243 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.428884  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.520376  8969 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 243 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:25.520478  8969 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:25.520591  8969 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 243 FOLLOWER]: Advancing to term 244
I1014 19:26:25.522836  8969 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.523380  8969 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 244 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.523954  8969 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 244 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:25.524838 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 244 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:25.524838 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 244 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:25.525050 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.525050 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.525583 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 244 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.525844 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 244 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.525931 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 244 election: Election decided. Result: candidate lost.
I1014 19:26:25.526234  8969 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:25.526314  8969 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Leader election lost for term 244. Reason: could not achieve majority
I1014 19:26:25.529660  9059 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:25.529753  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:25.529815  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 243 FOLLOWER]: Advancing to term 244
I1014 19:26:25.532039  9059 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 244 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.532577  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 244 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.533133 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 243. Current term is 244. Ops: []
I1014 19:26:25.533339  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 244 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:25.533433 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 244 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:25.533615 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 244: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:25.533879  9067 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 243.158, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I1014 19:26:25.533977 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 244 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 244: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:25.533955 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 244 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:25.534196 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.534310  8854 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Stepping down as leader of term 243
I1014 19:26:25.534426  8854 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:25.534644  8854 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 243.158, Last appended by leader: 158, Current term: 243, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.534843  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:25.534898  8854 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:25.535053  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 FOLLOWER]: Advancing to term 244
I1014 19:26:25.537276 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 244 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 243 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 244 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.537355 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 244 election: Election decided. Result: candidate lost.
I1014 19:26:25.538179  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:25.538273  9059 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 244 FOLLOWER]: Leader election lost for term 244. Reason: could not achieve majority
I1014 19:26:25.633081  8854 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 244 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:25.633179  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:25.633242  8854 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 244 FOLLOWER]: Advancing to term 245
I1014 19:26:25.635390  8854 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.635915  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 245 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.636132  8854 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 245 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.636785 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 245 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:25.636787 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 245 candidate_status { last_received { term: 243 index: 158 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:25.636934 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 244 FOLLOWER]: Advancing to term 245
I1014 19:26:25.636940 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 244 FOLLOWER]: Advancing to term 245
I1014 19:26:25.637013 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.218s (vote granted)
I1014 19:26:25.637019 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.152s (vote granted)
I1014 19:26:25.638993 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 245.
I1014 19:26:25.639098 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 245 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 245.
I1014 19:26:25.639365 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 245 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.639441 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 245 election: Election decided. Result: candidate won.
I1014 19:26:25.639690  8854 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:25.639748 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 245 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:25.639817  8854 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 FOLLOWER]: Leader election won for term 245
I1014 19:26:25.639924  8854 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:25.640106  8854 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 243.158, Last appended by leader: 158, Current term: 245, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.641252  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.641422  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.750943 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 245 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 243 index: 158. Preceding OpId from leader: term: 245 index: 159. (index mismatch)
I1014 19:26:25.751487  9067 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I1014 19:26:25.752801  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.753080  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.753937  8854 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.754150  8854 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.755100 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 243 index: 158. Preceding OpId from leader: term: 245 index: 159. (index mismatch)
I1014 19:26:25.755339  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.755551  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.756103  8854 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I1014 19:26:25.757692  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.757925  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:25.758081  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.758266  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:25.796650  8855 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.581s	user 0.002s	sys 0.000s
I1014 19:26:25.796715  8855 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:25.797559  8855 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:25.799439  8854 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 245 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:25.799748  8854 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:25.809089  8855 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:25.811233  8855 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:25.903772  8969 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 245 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:25.903880  8969 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:25.903957  8969 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 245 FOLLOWER]: Advancing to term 246
I1014 19:26:25.906147  8969 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.906764  8969 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 246 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:25.906986  8969 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 246 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:25.907559 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 246 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:25.907785 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 246 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.908076 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 246 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:25.908232 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 246 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 246 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.908345 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 246 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.908793 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 246 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 246 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:25.908874 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 246 election: Election decided. Result: candidate lost.
I1014 19:26:25.909238 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 245. Current term is 246. Ops: []
I1014 19:26:25.909416  8969 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:25.909569  8969 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Leader election lost for term 246. Reason: could not achieve majority
I1014 19:26:25.909802  9067 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 245.160, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I1014 19:26:25.910029  9067 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 LEADER]: Stepping down as leader of term 245
I1014 19:26:25.910084  9067 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:25.910259  9067 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 245.160, Last appended by leader: 160, Current term: 245, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:25.910465  9067 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:25.910524  9067 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:25.910652  9067 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 245 FOLLOWER]: Advancing to term 246
I1014 19:26:25.998272  9049 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:25.998373  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:25.998476  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 245 FOLLOWER]: Advancing to term 246
I1014 19:26:26.000672  9049 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.001190  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 246 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.001380  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 246 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.002051 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 246 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.002270 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 246: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:26.002727 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 246 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 246: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:26.006547 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 246 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.006744 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:26.008893 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 246 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 246.
I1014 19:26:26.009214 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 246 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.009289 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 246 election: Election decided. Result: candidate won.
I1014 19:26:26.009567  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:26.009673  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 FOLLOWER]: Leader election won for term 246
I1014 19:26:26.009898  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.010119  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 245.160, Last appended by leader: 160, Current term: 246, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.010869  8855 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:26.011335  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 246 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.011548  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.011803  8855 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:26.012806  8855 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 245 vs 246: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.013078  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.013309  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.013552  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.013779  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.014008  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.014225  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.014467  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 246 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.016366  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:26.018832  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:26.029083  9286 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:26.029178  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:26.029258  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 246 FOLLOWER]: Advancing to term 247
I1014 19:26:26.031409  9286 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.031950  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 247 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:26.032162  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 247 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.032771 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 247 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:26.032771 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 247 candidate_status { last_received { term: 245 index: 160 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.032953 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 246 FOLLOWER]: Advancing to term 247
I1014 19:26:26.032990 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 247 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.033057 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:26.033452 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 247 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 247 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.035277 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 247.
I1014 19:26:26.035755 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 247 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.035822 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 247 election: Election decided. Result: candidate won.
I1014 19:26:26.036131  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.171s (election complete)
I1014 19:26:26.036200  9286 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 FOLLOWER]: Leader election won for term 247
I1014 19:26:26.036300  9286 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:26.036556  9286 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 245.160, Last appended by leader: 160, Current term: 247, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.037842  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.038069  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.108364 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 246. Current term is 247. Ops: []
I1014 19:26:26.108973  9049 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I1014 19:26:26.109227  9059 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 LEADER]: Stepping down as leader of term 246
I1014 19:26:26.109279  9059 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.109544  9059 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 246.161, Last appended by leader: 161, Current term: 246, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.109695  9059 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:26.109755  9059 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:26.109864  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 246 FOLLOWER]: Advancing to term 247
I1014 19:26:26.117569 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 247 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 246 index: 161. Preceding OpId from leader: term: 247 index: 161. (term mismatch)
I1014 19:26:26.117632 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 160
I1014 19:26:26.117660 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 246.161
I1014 19:26:26.117722 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 247 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:26.118291  8969 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I1014 19:26:26.119807  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.120012  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.120885  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.121125  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.121495 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 245 index: 160. Preceding OpId from leader: term: 247 index: 161. (index mismatch)
I1014 19:26:26.122069  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.122153  9286 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I1014 19:26:26.122257  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.123653  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.123972  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.123765  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.124130  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.216341  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 247 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.216609  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.313705  9059 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 247 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:26.313802  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:26.313870  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 247 FOLLOWER]: Advancing to term 248
I1014 19:26:26.316047  9059 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.316591  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 248 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.316972  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 248 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.317593 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 248 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.317802 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 248 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.318174 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 248 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 248 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.318560 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 248 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.318763 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 248 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.319164 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 248 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 248 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.319234 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 248 election: Election decided. Result: candidate lost.
I1014 19:26:26.319574  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:26.319641  9059 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Leader election lost for term 248. Reason: could not achieve majority
I1014 19:26:26.334941 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 247. Current term is 248. Ops: []
I1014 19:26:26.335553  9469 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 247.161, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.335893  9286 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 LEADER]: Stepping down as leader of term 247
I1014 19:26:26.335965  9286 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:26.336225  9286 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 247.161, Last appended by leader: 161, Current term: 247, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.336465  9286 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:26.336539  9286 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:26.336694  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 247 FOLLOWER]: Advancing to term 248
I1014 19:26:26.400099  8919 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:26.400188  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:26.400261  8919 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 247 FOLLOWER]: Advancing to term 248
I1014 19:26:26.402436  8919 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.403020  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 248 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:26.403194  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 248 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.403949 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 248 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.404145 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:26.404192 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 248 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:26.404399 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 248: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:26.404832 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 248 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 248: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:26.406226 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 248 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 248.
I1014 19:26:26.406591 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 248 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.406666 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 248 election: Election decided. Result: candidate won.
I1014 19:26:26.406903  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:26.406988  8919 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 FOLLOWER]: Leader election won for term 248
I1014 19:26:26.407120  8919 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:26.407332  8919 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 247.161, Last appended by leader: 161, Current term: 248, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.409021  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 248 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.409227  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.409894  8855 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:26.411712  8855 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:26.453881  9059 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:26.453969  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:26.454041  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 248 FOLLOWER]: Advancing to term 249
I1014 19:26:26.456235  9059 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.456779  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 249 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.457463 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 249 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.457659 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 248 FOLLOWER]: Advancing to term 249
I1014 19:26:26.457741 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:26.457878  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 249 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.458429 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 249 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.458647 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 249 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.459039 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 249 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 249 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.459542 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 249 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 249.
I1014 19:26:26.459858 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 249 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.459924 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 249 election: Election decided. Result: candidate won.
I1014 19:26:26.460193  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.172s (election complete)
I1014 19:26:26.460263  9059 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 FOLLOWER]: Leader election won for term 249
I1014 19:26:26.460451  9059 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.460680  9059 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 247.161, Last appended by leader: 161, Current term: 249, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.462026  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 249 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.462357  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.495956 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 249 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 248. Current term is 249. Ops: []
I1014 19:26:26.496695  9067 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.496943  9067 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 LEADER]: Stepping down as leader of term 248
I1014 19:26:26.497001  9067 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:26.497185  9067 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 248.162, Last appended by leader: 162, Current term: 248, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.497395  9067 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:26.497475  9067 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:26.497618  9067 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 248 FOLLOWER]: Advancing to term 249
I1014 19:26:26.574453  9286 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 249 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:26.574584  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:26.574662  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 249 FOLLOWER]: Advancing to term 250
I1014 19:26:26.576597 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 249 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 248 index: 162. Preceding OpId from leader: term: 249 index: 162. (term mismatch)
I1014 19:26:26.576669 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 161
I1014 19:26:26.576704 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 248.162
I1014 19:26:26.576766 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 249 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:26.576887  9286 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.577291 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 249. Current term is 250. Ops: []
I1014 19:26:26.577360  9059 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.577479  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 250 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:26.577692  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 250 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.578148  9059 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.578236 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 250 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:26.578289 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 250 candidate_status { last_received { term: 247 index: 161 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.578398  9049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 LEADER]: Stepping down as leader of term 249
I1014 19:26:26.578475 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 249 FOLLOWER]: Advancing to term 250
I1014 19:26:26.578482  9049 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.578557 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:26.578675  9049 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 249.162, Last appended by leader: 162, Current term: 249, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.578891  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:26.578936  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:26.579035  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 249 FOLLOWER]: Advancing to term 250
I1014 19:26:26.580674 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 250 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 250.
I1014 19:26:26.581122 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 250 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 249. Current term is 250. Ops: []
I1014 19:26:26.581225 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 250 because replica has last-logged OpId of term: 249 index: 162, which is greater than that of the candidate, which has last-logged OpId of term: 247 index: 161.
I1014 19:26:26.581524 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 250 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.581672 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 250 election: Election decided. Result: candidate won.
I1014 19:26:26.581805 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 250 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 250 because replica has last-logged OpId of term: 249 index: 162, which is greater than that of the candidate, which has last-logged OpId of term: 247 index: 161.
I1014 19:26:26.582155  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:26.582262  9286 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 FOLLOWER]: Leader election won for term 250
I1014 19:26:26.582507  9286 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:26.582821  9286 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 247.161, Last appended by leader: 161, Current term: 250, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.584259  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.584514  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.669190 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 249 index: 162. Preceding OpId from leader: term: 250 index: 162. (term mismatch)
I1014 19:26:26.669255 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 161
I1014 19:26:26.669296 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 249.162
I1014 19:26:26.669348 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:26.670168  9286 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.671392  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.671629  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.680860 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 250 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 247 index: 161. Preceding OpId from leader: term: 250 index: 162. (index mismatch)
I1014 19:26:26.681560  9469 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I1014 19:26:26.681993  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.682220  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.683308  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.683452  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.683703  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.683727  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.688000  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.688231  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:26.782814  8919 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 250 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:26.782908  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:26.782984  8919 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 250 FOLLOWER]: Advancing to term 251
I1014 19:26:26.784766  8919 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.785295  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 251 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:26.785502  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 251 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.786375 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 251 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:26.786602 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 251 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.786691  8969 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 250 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.786944  8969 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.787014 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 250. Current term is 251. Ops: []
I1014 19:26:26.787173 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 251 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.787225 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 251 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 251 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.787349 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 251 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.787676 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 251 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 251 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.787761 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 251 election: Election decided. Result: candidate lost.
I1014 19:26:26.788010  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:26.788087  8919 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Leader election lost for term 251. Reason: could not achieve majority
I1014 19:26:26.788127  9286 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 250.162, Next index: 163, Last known committed idx: 162, Time since last communication: 0.000s
I1014 19:26:26.788353  9469 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 LEADER]: Stepping down as leader of term 250
I1014 19:26:26.788404  9469 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:26.788616  9469 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 250.162, Last appended by leader: 162, Current term: 250, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.788764  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:26.788826  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:26.788947  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 250 FOLLOWER]: Advancing to term 251
I1014 19:26:26.868378  9049 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:26.868510  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:26.868578  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 250 FOLLOWER]: Advancing to term 251
I1014 19:26:26.870707  9049 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.871203  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 251 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.872438 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 251 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.872639 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:26.874845 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 251 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 251.
I1014 19:26:26.875300  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 251 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:26.875805 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 251 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.875876 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 251 election: Election decided. Result: candidate won.
I1014 19:26:26.876046 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 251 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:26.876121  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:26.876195  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 FOLLOWER]: Leader election won for term 251
I1014 19:26:26.876241 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 251: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:26.876328  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.876673 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 251 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 251: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:26.876616  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 250.162, Last appended by leader: 162, Current term: 251, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.877746  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 251 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.877946  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.919137  8919 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:26.919243  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.154s (starting election)
I1014 19:26:26.919327  8919 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 251 FOLLOWER]: Advancing to term 252
I1014 19:26:26.921623  8919 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.922168  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 252 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:26.922401  8919 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 252 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.923046 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 252 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:26.923046 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 252 candidate_status { last_received { term: 250 index: 162 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:26.923209 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 251 FOLLOWER]: Advancing to term 252
I1014 19:26:26.923240 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 252 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.923301 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (vote granted)
I1014 19:26:26.925575 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 252.
I1014 19:26:26.926033 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 252 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:26.926110 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 252 election: Election decided. Result: candidate won.
I1014 19:26:26.926479  8919 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.168s (election complete)
I1014 19:26:26.926520 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 252 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 252 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:26.926625  8919 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 FOLLOWER]: Leader election won for term 252
I1014 19:26:26.926787  8919 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:26.927027  8919 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 250.162, Last appended by leader: 162, Current term: 252, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.928113  9067 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:26.928367  9067 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:26.980000 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 251. Current term is 252. Ops: []
I1014 19:26:26.980680  9059 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 163, Last known committed idx: 162, Time since last communication: 0.000s
I1014 19:26:26.980872  9059 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 252.163 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 162
I1014 19:26:26.980995  9059 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 LEADER]: Stepping down as leader of term 251
I1014 19:26:26.981045  9059 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:26.981230  9059 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 251.163, Last appended by leader: 163, Current term: 251, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:26.981429  9059 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:26.981488  9059 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:26.981631  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 251 FOLLOWER]: Advancing to term 252
W1014 19:26:26.984699  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.950s	user 0.002s	sys 0.000s
I1014 19:26:26.984755  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:26.985747  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:26.986975  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 241 vs 252: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.987392  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.987699  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.987978  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.988252  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.988523  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.988835  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.989114  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.989363  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.989656  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.989897  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.990165  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.990398  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.990684  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.990933  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.991217  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.994763  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.995077  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.995456  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.996117  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.996443  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.996809  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.997268  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.997711  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.997968  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:26.998242  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 252 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.039041 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 250 index: 162. Preceding OpId from leader: term: 252 index: 163. (index mismatch)
I1014 19:26:27.039641  9067 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 162, Time since last communication: 0.000s
I1014 19:26:27.040765  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.041103  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.041924  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.042183  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.042578 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 252 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 251 index: 163. Preceding OpId from leader: term: 252 index: 163. (term mismatch)
I1014 19:26:27.042634 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 162
I1014 19:26:27.042667 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 251.163
I1014 19:26:27.042712 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 252 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:27.042593  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.042776  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.044399  8919 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 162, Time since last communication: 0.000s
I1014 19:26:27.045720  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.045964  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.046308  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.046483  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.145479  9059 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 252 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:27.145598  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:27.145674  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 252 FOLLOWER]: Advancing to term 253
I1014 19:26:27.147918  9059 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.148411  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 253 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:27.148811  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 253 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.149173 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 253 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:27.149387 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 253 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.149459 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 253 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:27.149667 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 253 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.149762 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 253 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 253 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.150035 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 253 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 253 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.150107 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 253 election: Election decided. Result: candidate lost.
I1014 19:26:27.150326  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:27.150393  9059 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Leader election lost for term 253. Reason: could not achieve majority
I1014 19:26:27.163894  8919 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 252 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.164192  8919 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.164441 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 252. Current term is 253. Ops: []
I1014 19:26:27.164897  8919 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 252.163, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s
I1014 19:26:27.165141  9871 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Stepping down as leader of term 252
I1014 19:26:27.165196  9871 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:27.165418  9871 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 252.163, Last appended by leader: 163, Current term: 252, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.165622  9871 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:27.165676  9871 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:27.165788  9871 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 252 FOLLOWER]: Advancing to term 253
I1014 19:26:27.218958  9469 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:27.219060  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:27.219151  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 252 FOLLOWER]: Advancing to term 253
I1014 19:26:27.220894  9469 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.221431  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 253 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.221673  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 253 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.222193 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 253 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:27.222399 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 253: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:27.222717 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 253 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:27.222898 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:27.225096 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 253 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 253.
I1014 19:26:27.225503 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 253 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.225612 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 253 election: Election decided. Result: candidate won.
I1014 19:26:27.225736 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 253 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 253: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:27.225875  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:27.225981  9469 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 FOLLOWER]: Leader election won for term 253
I1014 19:26:27.226084  9469 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:27.226274  9469 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 252.163, Last appended by leader: 163, Current term: 253, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.227244  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 253 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.227545  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.227943  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:27.229815  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:27.280256  9059 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:27.280366  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.165s (starting election)
I1014 19:26:27.280444  9059 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 253 FOLLOWER]: Advancing to term 254
I1014 19:26:27.282533  9059 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.282969  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 254 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:27.283118  9059 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 254 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.283851 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 254 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:27.283851 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 254 candidate_status { last_received { term: 252 index: 163 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:27.284024 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 253 FOLLOWER]: Advancing to term 254
I1014 19:26:27.284055 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 254 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.284108 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:26:27.284627 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 254 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 254 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.286499 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 254 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 254.
I1014 19:26:27.287004 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 254 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.287066 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 254 election: Election decided. Result: candidate won.
I1014 19:26:27.287426  9059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:27.287497  9059 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 FOLLOWER]: Leader election won for term 254
I1014 19:26:27.287667  9059 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:27.287843  9059 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 252.163, Last appended by leader: 163, Current term: 254, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.288965  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.289146  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.342587 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 253. Current term is 254. Ops: []
I1014 19:26:27.343324  9286 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s
I1014 19:26:27.343583  9286 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 254.164 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 163
I1014 19:26:27.343686  9286 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 LEADER]: Stepping down as leader of term 253
I1014 19:26:27.343746  9286 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:27.343966  9286 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 253.164, Last appended by leader: 164, Current term: 253, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.344188  9286 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:27.344240  9286 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:27.344363  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 253 FOLLOWER]: Advancing to term 254
I1014 19:26:27.377063 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 254 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 252 index: 163. Preceding OpId from leader: term: 254 index: 164. (index mismatch)
I1014 19:26:27.377784  9049 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s
I1014 19:26:27.379246  9871 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.379564  9871 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.380816  9059 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.381018  9059 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.381417 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 253 index: 164. Preceding OpId from leader: term: 254 index: 164. (term mismatch)
I1014 19:26:27.381482 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 163
I1014 19:26:27.381541 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 253.164
I1014 19:26:27.381597 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:27.382176  9059 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s
I1014 19:26:27.383064  9871 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.383366  9871 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.384218  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.384511  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.384644  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.384821  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:27.428349  9330 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.409s	user 0.001s	sys 0.000s
I1014 19:26:27.428403  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:27.429466  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:27.432178 10017 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 254 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.432684 10017 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.446177  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:27.448542  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:27.614790  9871 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 254 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:27.614886  9871 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:27.614966  9871 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 254 FOLLOWER]: Advancing to term 255
I1014 19:26:27.617277  9871 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.617826  9871 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 255 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.618438  9871 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 255 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:27.618599 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 255 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:27.618803 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 255 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.619174 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 255 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 255 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.619346 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 255 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:27.619572 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 255 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.620118 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 255 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 255 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.620191 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 255 election: Election decided. Result: candidate lost.
I1014 19:26:27.620420  9871 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:27.620488  9871 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Leader election lost for term 255. Reason: could not achieve majority
I1014 19:26:27.623042 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 254. Current term is 255. Ops: []
I1014 19:26:27.623529 10023 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 254.165, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s
I1014 19:26:27.623842  9049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Stepping down as leader of term 254
I1014 19:26:27.623898  9049 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:27.624078  9049 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 254.165, Last appended by leader: 165, Current term: 254, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.624253  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:27.624315  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:27.624436  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 254 FOLLOWER]: Advancing to term 255
I1014 19:26:27.641556  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:27.642578  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:27.643776  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 254 vs 255: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 255 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:27.644073  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 255 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:27.644336  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 255 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:27.644606  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 255 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:27.644860  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 255 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.697772  9286 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:27.697890  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (starting election)
I1014 19:26:27.697970  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 254 FOLLOWER]: Advancing to term 255
I1014 19:26:27.700376  9286 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.700969  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 255 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.701385  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 255 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.702131 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 255 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:27.702316 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:27.703752 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 255 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:27.703959 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 255: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:27.704511 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 255 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 255: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:27.704638 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 255 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 255.
I1014 19:26:27.705004 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 255 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.705063 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 255 election: Election decided. Result: candidate won.
I1014 19:26:27.705302  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:27.705402  9286 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 FOLLOWER]: Leader election won for term 255
I1014 19:26:27.705548  9286 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:27.705782  9286 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 254.165, Last appended by leader: 165, Current term: 255, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.706794  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 255 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.707048  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.743386  9871 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:27.743494  9871 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.154s (starting election)
I1014 19:26:27.743573  9871 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 255 FOLLOWER]: Advancing to term 256
I1014 19:26:27.745694  9871 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.746202  9871 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 256 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.746454  9871 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 256 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:27.747067 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 256 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:27.747082 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 256 candidate_status { last_received { term: 254 index: 165 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:27.747284 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 256 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.747264 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 255 FOLLOWER]: Advancing to term 256
I1014 19:26:27.747349 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (vote granted)
I1014 19:26:27.747766 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 256 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 256 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.749565 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 256.
I1014 19:26:27.749949 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 256 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.750022 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 256 election: Election decided. Result: candidate won.
I1014 19:26:27.750558  9871 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.209s (election complete)
I1014 19:26:27.750661  9871 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 FOLLOWER]: Leader election won for term 256
I1014 19:26:27.751083  9871 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:27.751432  9871 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 254.165, Last appended by leader: 165, Current term: 256, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.752717 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.752964 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.785203 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 255. Current term is 256. Ops: []
I1014 19:26:27.785904  9469 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s
I1014 19:26:27.786175  9469 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 LEADER]: Stepping down as leader of term 255
I1014 19:26:27.786239  9469 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:27.786432  9469 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 255.166, Last appended by leader: 166, Current term: 255, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.786649  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:27.786713  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:27.786834  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 255 FOLLOWER]: Advancing to term 256
W1014 19:26:27.846848  8855 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.435s	user 0.001s	sys 0.001s
I1014 19:26:27.846923  8855 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:27.846930 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 254 index: 165. Preceding OpId from leader: term: 256 index: 166. (index mismatch)
I1014 19:26:27.847515 10192 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s
I1014 19:26:27.848227  8855 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:27.848927  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.849189  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.850783  9871 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.850975  9871 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.851189  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.851404  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.851902 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 256 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 255 index: 166. Preceding OpId from leader: term: 256 index: 167. (index mismatch)
I1014 19:26:27.852960  8855 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:27.852489 10202 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.001s
I1014 19:26:27.853294 10202 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 255.166 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 165
I1014 19:26:27.854374 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 165
I1014 19:26:27.854419 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 255.166
I1014 19:26:27.854502 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 256 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:27.855134  8855 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:27.855551  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.855792  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.856029  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.856284  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:27.955829  9286 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 256 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:27.955930  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:27.956010  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 256 FOLLOWER]: Advancing to term 257
I1014 19:26:27.957684  9286 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.958086  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 257 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:27.958287  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 257 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:27.959117 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 257 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:27.959125 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 257 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:27.959321 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 257 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.959347 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 257 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.959857 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 257 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 257 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.960201 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 257 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 257 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:27.960274 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 257 election: Election decided. Result: candidate lost.
I1014 19:26:27.960717  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:27.960793  9286 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Leader election lost for term 257. Reason: could not achieve majority
I1014 19:26:27.977529 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 256 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:27.977808 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:27.978971 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 256. Current term is 257. Ops: []
I1014 19:26:27.979614  9871 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 256.167, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I1014 19:26:27.979835  9871 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 LEADER]: Stepping down as leader of term 256
I1014 19:26:27.979893  9871 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:27.980082  9871 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 256.167, Last appended by leader: 167, Current term: 256, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:27.980288  9871 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:27.980341  9871 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:27.980515  9871 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 256 FOLLOWER]: Advancing to term 257
I1014 19:26:28.050617  9049 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:28.050712  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:28.050781  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 256 FOLLOWER]: Advancing to term 257
I1014 19:26:28.053019  9049 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.053555  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 257 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.053759  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 257 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.054229 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 257 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.054487 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 257: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:28.054432 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 257 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.054636 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:28.056818 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 257 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 257.
I1014 19:26:28.057235 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 257 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 257: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:28.057523 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 257 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.057587 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 257 election: Election decided. Result: candidate won.
I1014 19:26:28.058307  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:28.058378  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 FOLLOWER]: Leader election won for term 257
I1014 19:26:28.058508  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.058746  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 256.167, Last appended by leader: 167, Current term: 257, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.059864 10023 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 257 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.060097 10023 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.060293  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:28.061808  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:28.064564  9286 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.064657  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.197s (starting election)
I1014 19:26:28.064728  9286 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 257 FOLLOWER]: Advancing to term 258
I1014 19:26:28.066535  9286 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.066915  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 258 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.067122  9286 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 258 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.067751 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 258 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.067762 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 258 candidate_status { last_received { term: 256 index: 167 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.067906 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 257 FOLLOWER]: Advancing to term 258
I1014 19:26:28.067937 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 258 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.067993 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:28.068385 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 258 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 258 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.069635 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 258 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 258.
I1014 19:26:28.069937 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 258 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.069998 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 258 election: Election decided. Result: candidate won.
I1014 19:26:28.070271  9286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.182s (election complete)
I1014 19:26:28.070384  9286 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 FOLLOWER]: Leader election won for term 258
I1014 19:26:28.070542  9286 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:28.070739  9286 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 256.167, Last appended by leader: 167, Current term: 258, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.071676  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.071831  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.135723 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 257. Current term is 258. Ops: []
I1014 19:26:28.136451 10023 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I1014 19:26:28.136728 10023 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 258.168 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 167
I1014 19:26:28.136833  9049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 LEADER]: Stepping down as leader of term 257
I1014 19:26:28.136891  9049 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.137099  9049 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 257.168, Last appended by leader: 168, Current term: 257, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.137338  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:28.137398  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:28.137537  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 257 FOLLOWER]: Advancing to term 258
I1014 19:26:28.156245 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 257 index: 168. Preceding OpId from leader: term: 258 index: 168. (term mismatch)
I1014 19:26:28.156311 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 167
I1014 19:26:28.156338 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 257.168
I1014 19:26:28.156388 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:28.157073  9469 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I1014 19:26:28.158373  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.158635  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.160220  9286 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.160513  9286 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.160944  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.161155  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.161188 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 258 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 256 index: 167. Preceding OpId from leader: term: 258 index: 168. (index mismatch)
I1014 19:26:28.161715  9286 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I1014 19:26:28.163244 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.163244  9871 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.163661  9871 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.163780 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.201390  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:28.202088  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:28.204460 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.204794 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.206949  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:28.208851  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:28.240301  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:28.241052  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:28.241920  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 257 vs 258: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 258 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.307587 10192 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 258 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:28.307723 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:28.307796 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 258 FOLLOWER]: Advancing to term 259
I1014 19:26:28.310076 10192 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.310622 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 259 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.311002 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 259 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.311352 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 259 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.311566 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 259 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.311625 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 259 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.311856 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 259 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.311936 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 259 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 259 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.312218 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 259 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 259 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.312292 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 259 election: Election decided. Result: candidate lost.
I1014 19:26:28.312510 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:28.312578 10192 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Leader election lost for term 259. Reason: could not achieve majority
I1014 19:26:28.328212 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 258. Current term is 259. Ops: []
I1014 19:26:28.328780 10350 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 258.169, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.329010  9469 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Stepping down as leader of term 258
I1014 19:26:28.329064  9469 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:28.329285  9469 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 258.169, Last appended by leader: 169, Current term: 258, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.329440  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:28.329520  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:28.329623  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 258 FOLLOWER]: Advancing to term 259
I1014 19:26:28.386359  9049 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:28.386477  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:28.386551  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 258 FOLLOWER]: Advancing to term 259
I1014 19:26:28.388067  9049 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.388566  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 259 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.388754  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 259 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.389269 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 259 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.389365 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 259 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.389456 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:28.389593 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 259: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:28.390967 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 259 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 259.
I1014 19:26:28.391276 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 259 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 259: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:28.391428 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 259 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.391521 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 259 election: Election decided. Result: candidate won.
I1014 19:26:28.391747  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:28.391815  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 FOLLOWER]: Leader election won for term 259
I1014 19:26:28.391952  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.392177  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 258.169, Last appended by leader: 169, Current term: 259, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.393050 10023 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 259 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.393306 10023 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.393532  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:28.395602  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:28.446414 10192 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.446555 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.179s (starting election)
I1014 19:26:28.446632 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 259 FOLLOWER]: Advancing to term 260
I1014 19:26:28.449053 10192 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.450145 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 260 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.451268 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 260 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.451534 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 260 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.451611 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 260 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.452303 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 260 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.452523 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 259 FOLLOWER]: Advancing to term 260
I1014 19:26:28.452618 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 260 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 260 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.452641 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (vote granted)
I1014 19:26:28.454836 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 260 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 260.
I1014 19:26:28.455488 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 260 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.455570 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 260 election: Election decided. Result: candidate won.
I1014 19:26:28.455812 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:28.455900 10192 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 FOLLOWER]: Leader election won for term 260
I1014 19:26:28.456024 10192 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:28.456264 10192 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 258.169, Last appended by leader: 169, Current term: 260, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.457307 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 260 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.457661 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.478032 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 260 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 259. Current term is 260. Ops: []
I1014 19:26:28.478392 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 259. Current term is 260. Ops: []
I1014 19:26:28.479185  9049 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.485337  9049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 260.170 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 169
I1014 19:26:28.486155 10023 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.486485 10480 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 LEADER]: Stepping down as leader of term 259
I1014 19:26:28.486582 10480 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.490109 10480 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 259.170, Last appended by leader: 170, Current term: 259, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.490404 10480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:28.490526 10480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:28.490741 10480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 259 FOLLOWER]: Advancing to term 260
I1014 19:26:28.559629  9469 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 260 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.559710  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:28.559778  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 260 FOLLOWER]: Advancing to term 261
I1014 19:26:28.562001  9469 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 261 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.562506  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 261 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.562672  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 261 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.563414 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 261 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.563657 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 261 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.564302 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 261 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 261 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.564411 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 261 candidate_status { last_received { term: 258 index: 169 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.564616 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 260 FOLLOWER]: Advancing to term 261
I1014 19:26:28.566210 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 261 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 261 because replica has last-logged OpId of term: 259 index: 170, which is greater than that of the candidate, which has last-logged OpId of term: 258 index: 169.
I1014 19:26:28.566582 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 261 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 261 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 261 because replica has last-logged OpId of term: 259 index: 170, which is greater than that of the candidate, which has last-logged OpId of term: 258 index: 169.
I1014 19:26:28.566651 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 261 election: Election decided. Result: candidate lost.
I1014 19:26:28.567065  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:28.567165  9469 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 261 FOLLOWER]: Leader election lost for term 261. Reason: could not achieve majority
I1014 19:26:28.571256 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 261 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 260. Current term is 261. Ops: []
I1014 19:26:28.571902  9871 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.572093  9871 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 259.170 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 169
I1014 19:26:28.572170 10192 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Stepping down as leader of term 260
I1014 19:26:28.572232 10192 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:28.572448 10192 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 260.170, Last appended by leader: 170, Current term: 260, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.572640 10192 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:28.572697 10192 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:28.572824 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 260 FOLLOWER]: Advancing to term 261
I1014 19:26:28.591184 10480 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 261 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.591277 10480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:28.591361 10480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 261 FOLLOWER]: Advancing to term 262
I1014 19:26:28.593621 10480 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.594077 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 262 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.594730 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 262 candidate_status { last_received { term: 259 index: 170 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.594897 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 261 FOLLOWER]: Advancing to term 262
I1014 19:26:28.594977 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.211s (vote granted)
I1014 19:26:28.597271 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 262 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 262.
I1014 19:26:28.597668 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 262 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.597743 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 262 election: Election decided. Result: candidate won.
I1014 19:26:28.597947  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:28.598021  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 FOLLOWER]: Leader election won for term 262
I1014 19:26:28.598120  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.598320  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 259.170, Last appended by leader: 170, Current term: 262, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.599285  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 262 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.599643 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 262 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.599720  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.600328 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 262 candidate_status { last_received { term: 259 index: 170 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.600536 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 261 FOLLOWER]: Advancing to term 262
I1014 19:26:28.602763 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 262 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 262 because replica has last-logged OpId of term: 260 index: 170, which is greater than that of the candidate, which has last-logged OpId of term: 259 index: 170.
I1014 19:26:28.603168 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 262 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 262 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 262 because replica has last-logged OpId of term: 260 index: 170, which is greater than that of the candidate, which has last-logged OpId of term: 259 index: 170.
I1014 19:26:28.651621 10192 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 262 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.651721 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:28.651790 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 262 FOLLOWER]: Advancing to term 263
I1014 19:26:28.654034 10192 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.654556 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 263 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.654732 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 263 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.655527 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 263 candidate_status { last_received { term: 260 index: 170 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.655536 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 263 candidate_status { last_received { term: 260 index: 170 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.655730 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 263 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.655715 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 262 FOLLOWER]: Advancing to term 263
I1014 19:26:28.655798 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.219s (vote granted)
I1014 19:26:28.656154 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 263 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 263 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.657932 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 263 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 263.
I1014 19:26:28.658257 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 263 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.658320 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 263 election: Election decided. Result: candidate won.
I1014 19:26:28.658670 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:28.658747 10192 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 FOLLOWER]: Leader election won for term 263
I1014 19:26:28.658892 10192 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:28.659142 10192 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 260.170, Last appended by leader: 170, Current term: 263, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.660267  9871 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.660599  9871 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.704740 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 262. Current term is 263. Ops: []
I1014 19:26:28.705267  9049 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.705509  9049 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 263.171 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 169
I1014 19:26:28.705901 10480 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 LEADER]: Stepping down as leader of term 262
I1014 19:26:28.705994 10480 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.706208 10480 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 262.171, Last appended by leader: 171, Current term: 262, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.706364 10480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:28.706421 10480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:28.706542 10480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 262 FOLLOWER]: Advancing to term 263
I1014 19:26:28.747872 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 263 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 258 index: 169. Preceding OpId from leader: term: 263 index: 171. (index mismatch)
I1014 19:26:28.748430  9871 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.749667  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.749949  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.751597 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.751806 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.752190 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.752476 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.752282  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.752594  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.753249 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 262 index: 171. Preceding OpId from leader: term: 263 index: 171. (term mismatch)
I1014 19:26:28.753320 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 170
I1014 19:26:28.753350 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 262.171
I1014 19:26:28.753406 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:28.753919 10192 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s
I1014 19:26:28.754051 10192 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 259.170 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 169
I1014 19:26:28.754683 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 169
I1014 19:26:28.754731 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 259.170
I1014 19:26:28.754778 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:28.756142 10023 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.756355 10023 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.756672  9049 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.756672 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.756992 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.756989  9049 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:28.837260 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.837548 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.851694  9469 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 263 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:28.851800  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:28.851876  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 263 FOLLOWER]: Advancing to term 264
I1014 19:26:28.854311  9469 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.854872  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 264 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.855093  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 264 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.855823 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 264 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:28.856016 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 264 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.856472 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 264 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 264 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.856403 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 264 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.856705 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 264 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.857215 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 264 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 264 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:28.857304 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 264 election: Election decided. Result: candidate lost.
I1014 19:26:28.857587  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:28.857688  9469 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Leader election lost for term 264. Reason: could not achieve majority
I1014 19:26:28.872153 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 263. Current term is 264. Ops: []
I1014 19:26:28.872795  9871 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 263.171, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I1014 19:26:28.873037 10192 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Stepping down as leader of term 263
I1014 19:26:28.873100 10192 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:28.873309 10192 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 263.171, Last appended by leader: 171, Current term: 263, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.873533 10192 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:28.873589 10192 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:28.873736 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 263 FOLLOWER]: Advancing to term 264
I1014 19:26:28.879899  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:28.880903  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:28.882750  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 259 vs 263: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:28.883138  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 263 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.939596  9049 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:28.939687  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:28.939770  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 263 FOLLOWER]: Advancing to term 264
I1014 19:26:28.941290  9049 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.941711  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 264 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:28.941877  9049 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 264 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.942479 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 264 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:28.942587 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 264 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.942703 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 264: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:28.942768 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:28.943181 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 264 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 264: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:28.944633 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 264 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 264.
I1014 19:26:28.945034 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 264 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.945098 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 264 election: Election decided. Result: candidate won.
I1014 19:26:28.945466  9049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:28.945546  9049 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 FOLLOWER]: Leader election won for term 264
I1014 19:26:28.945672  9049 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:28.945870  9049 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 263.171, Last appended by leader: 171, Current term: 264, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.948025 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 264 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:28.948274 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:28.948664  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:28.950259  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:28.996639  9469 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:28.996726  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.178s (starting election)
I1014 19:26:28.996803  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 264 FOLLOWER]: Advancing to term 265
I1014 19:26:28.998438  9469 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:28.998839  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 265 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:28.998998  9469 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 265 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:28.999797 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 265 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:28.999990 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 264 FOLLOWER]: Advancing to term 265
I1014 19:26:28.999992 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 265 candidate_status { last_received { term: 263 index: 171 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.000082 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:29.000196 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 265 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.000622 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 265 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 265 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.001765 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 265 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 265.
I1014 19:26:29.002132 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 265 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.002200 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 265 election: Election decided. Result: candidate won.
I1014 19:26:29.002568  9469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.197s (election complete)
I1014 19:26:29.002640  9469 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 FOLLOWER]: Leader election won for term 265
I1014 19:26:29.002744  9469 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:29.002948  9469 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 263.171, Last appended by leader: 171, Current term: 265, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.003911  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.004194  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.041116 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 265 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 264. Current term is 265. Ops: []
I1014 19:26:29.041766  9049 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I1014 19:26:29.042030  9049 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 LEADER]: Stepping down as leader of term 264
I1014 19:26:29.042088  9049 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:29.042328  9049 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 264.172, Last appended by leader: 172, Current term: 264, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.042570  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:29.042627  9049 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:29.042773  9049 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 264 FOLLOWER]: Advancing to term 265
I1014 19:26:29.042815  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:29.043817  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:29.047626 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 265 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 263 index: 171. Preceding OpId from leader: term: 265 index: 173. (index mismatch)
I1014 19:26:29.047819 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 265 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 264 index: 172. Preceding OpId from leader: term: 265 index: 173. (index mismatch)
I1014 19:26:29.048264  9469 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I1014 19:26:29.048754  9469 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I1014 19:26:29.048894  9469 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 264.172 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 171
I1014 19:26:29.049614 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 171
I1014 19:26:29.049671 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 264.172
I1014 19:26:29.049733 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 265 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:29.049929 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.050292 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.050734 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.050943 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.052480  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:29.053025 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.053324 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.052475 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.053723 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.053732 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.053923 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.053897 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.054433 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.056419  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:29.205135  8855 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.350s	user 0.001s	sys 0.000s
I1014 19:26:29.205190  8855 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:29.206148  8855 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:29.207193  8855 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 256 vs 265: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.207881  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.208154  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.208511  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.208773  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.209024  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.209280  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.209578  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.209825  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:29.210085  8855 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 265 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.230268 10192 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 265 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:29.230370 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:29.230468 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 265 FOLLOWER]: Advancing to term 266
I1014 19:26:29.232720 10192 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.233248 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 266 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.235316 10480 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 265 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:29.235389 10480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:29.235528 10480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 265 FOLLOWER]: Advancing to term 266
I1014 19:26:29.235967 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 266 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.236809 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 266 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:29.237046 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 266 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.237424 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 266 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 266 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.237346 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 266 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.237699 10480 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 266 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.238049 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 266 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 266: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:29.238257 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 266 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.238423 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 266 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 266 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 266: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:29.238531 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 266 election: Election decided. Result: candidate lost.
I1014 19:26:29.238400 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 266 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.239179 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 266 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:29.239192 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 266 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:29.239341 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 266 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.239368 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 266: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:29.239689 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:29.239753 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 266 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 266 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.239858 10192 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Leader election lost for term 266. Reason: could not achieve majority
I1014 19:26:29.239869 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 266 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 266: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:29.239939 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 266 election: Election decided. Result: candidate lost.
I1014 19:26:29.240675 10480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (election complete)
I1014 19:26:29.240752 10480 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 266 FOLLOWER]: Leader election lost for term 266. Reason: could not achieve majority
I1014 19:26:29.242584 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 265. Current term is 266. Ops: []
I1014 19:26:29.243079  9469 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 265.173, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I1014 19:26:29.243314  9469 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Stepping down as leader of term 265
I1014 19:26:29.243362  9469 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:29.243580  9469 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 173, Committed index: 173, Last appended: 265.173, Last appended by leader: 173, Current term: 265, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.243757  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:29.243815  9469 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:29.243944  9469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 265 FOLLOWER]: Advancing to term 266
I1014 19:26:29.348716 10668 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 266 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:29.348819 10668 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:29.348891 10668 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 266 FOLLOWER]: Advancing to term 267
I1014 19:26:29.351060 10668 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.351606 10668 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 267 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.352236 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 267 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.352340 10668 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 267 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.352403 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 266 FOLLOWER]: Advancing to term 267
I1014 19:26:29.352519 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.216s (vote granted)
I1014 19:26:29.352957 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 267 candidate_status { last_received { term: 265 index: 173 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:29.353116 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 266 FOLLOWER]: Advancing to term 267
I1014 19:26:29.353194 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.157s (vote granted)
I1014 19:26:29.354578 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 267 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 267.
I1014 19:26:29.355038 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 267 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.355104 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 267 election: Election decided. Result: candidate won.
I1014 19:26:29.355279 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 267 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 267.
I1014 19:26:29.355304 10668 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:29.355370 10668 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 FOLLOWER]: Leader election won for term 267
I1014 19:26:29.355494 10668 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:29.356075 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 267 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.355933 10668 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 173, Committed index: 173, Last appended: 265.173, Last appended by leader: 173, Current term: 267, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.357043  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.357265  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.480664 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 267 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 265 index: 173. Preceding OpId from leader: term: 267 index: 174. (index mismatch)
I1014 19:26:29.481770 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 267 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 265 index: 173. Preceding OpId from leader: term: 267 index: 174. (index mismatch)
I1014 19:26:29.481683 10668 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I1014 19:26:29.482537 10668 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 174, Last known committed idx: 173, Time since last communication: 0.000s
I1014 19:26:29.482975 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.483209 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.484534  9469 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.484511 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.484763 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.484760  9469 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.485011 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.485298 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.485622 10668 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.485831 10668 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.486711 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 267 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.486917 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.585273 10480 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 267 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:29.585367 10480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:29.585464 10480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 267 FOLLOWER]: Advancing to term 268
I1014 19:26:29.587601 10480 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.587972 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 267. Current term is 268. Ops: []
I1014 19:26:29.588208 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 268 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.588438 10480 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 268 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.588701 10192 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 267 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:29.588810 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:29.588887 10192 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 267 FOLLOWER]: Advancing to term 268
I1014 19:26:29.588919 10668 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 267.174, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I1014 19:26:29.589172 10668 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 LEADER]: Stepping down as leader of term 267
I1014 19:26:29.589239 10668 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:29.589463 10668 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 267.174, Last appended by leader: 174, Current term: 267, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.589710 10668 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:29.589766 10668 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:29.589912 10668 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 267 FOLLOWER]: Advancing to term 268
I1014 19:26:29.591054 10192 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.591547 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 268 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.591714 10192 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 268 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.591969 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 268 candidate_status { last_received { term: 267 index: 174 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:29.592144 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 268: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:29.592097 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 268 candidate_status { last_received { term: 267 index: 174 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:29.592273 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:26:29.593278 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 268 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 268: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:29.593660 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 268 candidate_status { last_received { term: 267 index: 174 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:29.594722 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 268 candidate_status { last_received { term: 267 index: 174 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.594885 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 268: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:29.595247 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 268.
I1014 19:26:29.595423 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 268 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:29.595674 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 268 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.595760 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 268 election: Election decided. Result: candidate won.
I1014 19:26:29.595752 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 268 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 268: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:29.595928 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 268 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 268 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:29.596009 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 268 election: Election decided. Result: candidate lost.
I1014 19:26:29.596033 10480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:29.596169 10480 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Leader election won for term 268
I1014 19:26:29.596287 10192 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:29.596374 10192 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Leader election lost for term 268. Reason: could not achieve majority
I1014 19:26:29.605144 10480 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:29.605417 10480 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 267.174, Last appended by leader: 174, Current term: 268, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.606921 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.607187 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.692158 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 267 index: 174. Preceding OpId from leader: term: 268 index: 175. (index mismatch)
I1014 19:26:29.692766 10946 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I1014 19:26:29.694133 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.694511 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.695542 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.695777 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.696007 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.696323 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.697535 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 267 index: 174. Preceding OpId from leader: term: 268 index: 175. (index mismatch)
I1014 19:26:29.698092 10480 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I1014 19:26:29.707015 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.707345 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.707015 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.707623 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:29.776109 10480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 268 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.776347 10480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.878381 10350 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:29.878487 10350 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:29.878583 10350 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 268 FOLLOWER]: Advancing to term 269
I1014 19:26:29.880743 10350 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.881211 10350 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 269 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.881448 10350 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 269 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:29.881974 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 269 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.882046 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 269 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:29.882190 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 269 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.882247 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 269 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.882748 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 269 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 269 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.883877 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 269 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 269 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:29.884083 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 269 election: Election decided. Result: candidate lost.
I1014 19:26:29.884367 10350 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:29.884469 10350 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Leader election lost for term 269. Reason: could not achieve majority
I1014 19:26:29.901017 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 268. Current term is 269. Ops: []
I1014 19:26:29.901628 10951 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 268.175, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I1014 19:26:29.901932 10946 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 LEADER]: Stepping down as leader of term 268
I1014 19:26:29.902000 10946 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:29.902185 10946 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 268.175, Last appended by leader: 175, Current term: 268, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.902385 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:29.902462 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:29.902637 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 268 FOLLOWER]: Advancing to term 269
I1014 19:26:29.965885 10950 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:29.965998 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:29.966078 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 268 FOLLOWER]: Advancing to term 269
I1014 19:26:29.968302 10950 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.968845 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 269 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.969019 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 269 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:29.969879 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 269 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:29.969996 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 269 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:29.970059 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (vote granted)
I1014 19:26:29.970192 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 269: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:29.970675 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 269 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 269: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:29.971967 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 269 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 269.
I1014 19:26:29.972359 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 269 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:29.972429 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 269 election: Election decided. Result: candidate won.
I1014 19:26:29.972812 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:29.972918 10950 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 FOLLOWER]: Leader election won for term 269
I1014 19:26:29.973171 10950 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:29.973502 10950 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 268.175, Last appended by leader: 175, Current term: 269, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:29.974503 10192 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 269 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:29.974757 10192 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:29.975281 11048 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:29.977455 11048 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:30.026641 10350 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:30.026751 10350 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:30.026818 10350 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 269 FOLLOWER]: Advancing to term 270
I1014 19:26:30.028280 10350 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.028715 10350 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 270 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:30.028903 10350 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 270 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:30.029460 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 270 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:30.029489 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 270 candidate_status { last_received { term: 268 index: 175 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:30.029676 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 270 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:30.029649 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 269 FOLLOWER]: Advancing to term 270
I1014 19:26:30.029737 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (vote granted)
I1014 19:26:30.030015 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 270 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 270 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:30.031860 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 270 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 270.
I1014 19:26:30.032259 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 270 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:30.032337 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 270 election: Election decided. Result: candidate won.
I1014 19:26:30.032596 10350 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.222s (election complete)
I1014 19:26:30.032681 10350 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 FOLLOWER]: Leader election won for term 270
I1014 19:26:30.032825 10350 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:30.033042 10350 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 268.175, Last appended by leader: 175, Current term: 270, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.034124 10668 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.034281 10668 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.072254 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 269. Current term is 270. Ops: []
I1014 19:26:30.072908 10192 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I1014 19:26:30.073184 10192 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 270.176 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 175
I1014 19:26:30.073316 10950 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 LEADER]: Stepping down as leader of term 269
I1014 19:26:30.073382 10950 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:30.073617 10950 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 269.176, Last appended by leader: 176, Current term: 269, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.073793 10950 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:30.073863 10950 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:30.073993 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 269 FOLLOWER]: Advancing to term 270
I1014 19:26:30.116827 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 270 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 269 index: 176. Preceding OpId from leader: term: 270 index: 176. (term mismatch)
I1014 19:26:30.116906 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 175
I1014 19:26:30.116940 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 269.176
I1014 19:26:30.116997 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 270 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:30.117765 10668 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I1014 19:26:30.119122 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.119513 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.120373 10350 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.120615 10350 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.120903 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 270 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 268 index: 175. Preceding OpId from leader: term: 270 index: 176. (index mismatch)
I1014 19:26:30.121021 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.121268 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.121752 10668 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I1014 19:26:30.123111 10951 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.123684 10951 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.123270 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.123807 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:30.180295  9330 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.230s	user 0.002s	sys 0.000s
I1014 19:26:30.180380  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:30.181350  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:30.182569  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 264 vs 270: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:30.182912  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:30.183187  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:30.183420  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.215482 10668 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 270 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.215811 10668 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:30.413421  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.357s	user 0.001s	sys 0.000s
I1014 19:26:30.413527  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:30.414518  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:30.430619  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:30.432785  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:30.520000 10946 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 270 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:30.520107 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:30.520179 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 270 FOLLOWER]: Advancing to term 271
I1014 19:26:30.522435 10946 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.522938 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 270. Current term is 271. Ops: []
I1014 19:26:30.523059 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 271 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.523278 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 271 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:30.523512 11128 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 270.177, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I1014 19:26:30.523880 10950 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 270 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:30.523952 11128 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 LEADER]: Stepping down as leader of term 270
I1014 19:26:30.523994 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:30.524135 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 270 FOLLOWER]: Advancing to term 271
I1014 19:26:30.524147 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 271 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:30.524258 11128 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:30.524458 11128 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 270.177, Last appended by leader: 177, Current term: 270, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.524722 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 271 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:30.524929 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:30.524986 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:30.525128 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 270 FOLLOWER]: Advancing to term 271
I1014 19:26:30.526250 10950 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.526664 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 271: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:30.526829 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 271 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:30.527072 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 271 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 271: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:30.527192 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 271 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.527240 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:30.527377 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 271 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:30.527577 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 271: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:30.527762 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 271 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:30.528067 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 271 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 271: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:30.529376 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 271.
I1014 19:26:30.529611 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 271 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:30.529867 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 271 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.529942 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 271 election: Election decided. Result: candidate won.
I1014 19:26:30.529985 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 271 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 271 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:30.530059 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 271 election: Election decided. Result: candidate lost.
I1014 19:26:30.530342 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:30.530438 10950 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Leader election lost for term 271. Reason: could not achieve majority
I1014 19:26:30.530566 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:30.530648 10946 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Leader election won for term 271
I1014 19:26:30.530783 10946 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:30.530979 10946 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 270.177, Last appended by leader: 177, Current term: 271, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.532004 10951 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 271 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.532246 10951 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.532464  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:30.534140  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:30.628705 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 270 index: 177. Preceding OpId from leader: term: 271 index: 178. (index mismatch)
I1014 19:26:30.629305 10951 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I1014 19:26:30.630530 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 271 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.630861 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.631906 10951 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 271 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.632141 10950 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:30.632251 10951 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.632256 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.222s (starting election)
I1014 19:26:30.632366 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 271 FOLLOWER]: Advancing to term 272
I1014 19:26:30.633421 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 271 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.633682 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.634444 10950 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.634810 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 271. Current term is 272. Ops: []
I1014 19:26:30.634995 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 272 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:30.635221 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 272 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.635746 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 272 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:30.635913 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 272 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:30.636229 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 272 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 272 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:30.639523 10951 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.004s
I1014 19:26:30.639552 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 272 candidate_status { last_received { term: 270 index: 177 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:30.639824 10946 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 LEADER]: Stepping down as leader of term 271
I1014 19:26:30.640014 10946 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:30.640249 10946 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 271.178, Last appended by leader: 178, Current term: 271, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.640518 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:30.640601 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:30.640799 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 271 FOLLOWER]: Advancing to term 272
I1014 19:26:30.643066 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 272 because replica has last-logged OpId of term: 271 index: 178, which is greater than that of the candidate, which has last-logged OpId of term: 270 index: 177.
I1014 19:26:30.643528 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 272 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 272 because replica has last-logged OpId of term: 271 index: 178, which is greater than that of the candidate, which has last-logged OpId of term: 270 index: 177.
I1014 19:26:30.643635 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 272 election: Election decided. Result: candidate lost.
I1014 19:26:30.643884 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.179s (election complete)
I1014 19:26:30.643956 10950 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Leader election lost for term 272. Reason: could not achieve majority
I1014 19:26:30.733863 11128 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:30.733976 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:30.734046 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 271 FOLLOWER]: Advancing to term 272
I1014 19:26:30.736187 11128 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 272 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.736538 10946 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 272 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:30.736631 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:30.736698 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 272 FOLLOWER]: Advancing to term 273
I1014 19:26:30.736737 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 272 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:30.736932 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 272 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:30.737591 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 272 candidate_status { last_received { term: 271 index: 178 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:30.739081 10946 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.739498 28639 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 272. Current term is 273.
W1014 19:26:30.740031 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 272 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 272. Current term is 273.
I1014 19:26:30.740073 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 272 election: Cancelling election due to peer responding with higher term
I1014 19:26:30.740262 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:30.740341 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 272 FOLLOWER]: Advancing to term 273
I1014 19:26:30.741552 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 272 candidate_status { last_received { term: 271 index: 178 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:30.741775 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 272: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:30.742251 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 272 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 272: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:30.742389 11128 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Leader election lost for term 272. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 272. Current term is 273.
I1014 19:26:30.743222 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 273 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.743827 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 273 candidate_status { last_received { term: 271 index: 178 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:30.743991 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.156s (vote granted)
I1014 19:26:30.745960 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 273.
I1014 19:26:30.746348 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 273 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:30.746424 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 273 election: Election decided. Result: candidate won.
I1014 19:26:30.746739 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 273 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:30.746749 11291 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:30.746906 11291 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Leader election won for term 273
I1014 19:26:30.747102 11291 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:30.747443 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 273 candidate_status { last_received { term: 271 index: 178 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:30.747334 11291 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 271.178, Last appended by leader: 178, Current term: 273, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:30.747601 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 272 FOLLOWER]: Advancing to term 273
I1014 19:26:30.747689 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
I1014 19:26:30.749825 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 273 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 273.
I1014 19:26:30.750222 10951 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.750457 10951 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.750707 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 273 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:30.851111 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 271 index: 178. Preceding OpId from leader: term: 273 index: 179. (index mismatch)
I1014 19:26:30.851714 10946 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I1014 19:26:30.853099 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 273 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 270 index: 177. Preceding OpId from leader: term: 273 index: 179. (index mismatch)
I1014 19:26:30.853026 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.853226 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.853669 10951 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 177, Time since last communication: 0.000s
I1014 19:26:30.854526 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.854724 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.856029 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.856199 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.860888 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.861222 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.861268 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.861580 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:30.862140 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.862462 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:30.861896 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 273 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:30.863114 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.056615 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 273 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:31.056718 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:31.056820 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 273 FOLLOWER]: Advancing to term 274
I1014 19:26:31.059123 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.059669 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 274 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.060133 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 274 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:31.060737 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 274 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.060945 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 274 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.061408 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 274 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 274 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.061669 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 274 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:31.061870 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 274 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.062337 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 274 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 274 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.062418 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 274 election: Election decided. Result: candidate lost.
I1014 19:26:31.062700 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:31.062836 11419 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Leader election lost for term 274. Reason: could not achieve majority
I1014 19:26:31.065934 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 273. Current term is 274. Ops: []
I1014 19:26:31.066467 11291 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 273.179, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I1014 19:26:31.066690 10946 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 LEADER]: Stepping down as leader of term 273
I1014 19:26:31.066749 10946 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:31.066951 10946 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 273.179, Last appended by leader: 179, Current term: 273, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.067142 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:31.067209 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:31.067350 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 273 FOLLOWER]: Advancing to term 274
I1014 19:26:31.071919  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:31.073123  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:31.074242  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 271 vs 274: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 274 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.074550  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 274 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.074810  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 274 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.075050  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 274 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.075361  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 274 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.128602 11128 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:31.128720 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (starting election)
I1014 19:26:31.128788 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 273 FOLLOWER]: Advancing to term 274
I1014 19:26:31.130731 11128 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.131294 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 274 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.131494 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 274 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:31.132144 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 274 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.132328 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (vote granted)
I1014 19:26:31.132364 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 274 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:31.132562 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 274: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:31.132925 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 274 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 274: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:31.134361 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 274 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 274.
I1014 19:26:31.134735 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 274 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.134796 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 274 election: Election decided. Result: candidate won.
I1014 19:26:31.135047 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:31.135130 11128 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 FOLLOWER]: Leader election won for term 274
I1014 19:26:31.136664 11128 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:31.136939 11128 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 273.179, Last appended by leader: 179, Current term: 274, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.138334 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 274 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.138586 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.174110 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:31.174201 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:31.174283 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 274 FOLLOWER]: Advancing to term 275
I1014 19:26:31.176512 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.177008 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 275 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.177209 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 275 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:31.177851 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 275 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.177901 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 275 candidate_status { last_received { term: 273 index: 179 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:31.178020 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 274 FOLLOWER]: Advancing to term 275
I1014 19:26:31.178112 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (vote granted)
I1014 19:26:31.178138 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 275 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.178561 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 275 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 275 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.180261 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 275.
I1014 19:26:31.180569 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 275 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.180654 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 275 election: Election decided. Result: candidate won.
I1014 19:26:31.180930 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:31.180999 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 FOLLOWER]: Leader election won for term 275
I1014 19:26:31.181161 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:31.181345 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 273.179, Last appended by leader: 179, Current term: 275, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.182323 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.182605 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.239328 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 274. Current term is 275. Ops: []
I1014 19:26:31.240329 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I1014 19:26:31.240626 11128 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 LEADER]: Stepping down as leader of term 274
I1014 19:26:31.240681 11128 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:31.240882 11128 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 274.180, Last appended by leader: 180, Current term: 274, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.241044 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:31.241113 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:31.241219 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 274 FOLLOWER]: Advancing to term 275
I1014 19:26:31.243738  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:31.244652  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:31.246057  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 270 vs 275: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.246320  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.246767  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.247009  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.247251  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.247515  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.247759  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.248013  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.248319  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.248646  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.248908  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:31.249157  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 275 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.280812 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 273 index: 179. Preceding OpId from leader: term: 275 index: 180. (index mismatch)
I1014 19:26:31.281342 10950 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I1014 19:26:31.282640 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.282882 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.283910 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 275 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 274 index: 180. Preceding OpId from leader: term: 275 index: 180. (term mismatch)
I1014 19:26:31.283979 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 179
I1014 19:26:31.284014 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 274.180
I1014 19:26:31.284081 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 275 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:31.284669 11419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I1014 19:26:31.286346 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.286341 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.286648 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.286718 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.297821 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.298151 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.298344 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.298588 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.386728 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 275 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:31.386850 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:31.386920 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 275 FOLLOWER]: Advancing to term 276
I1014 19:26:31.388937 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.389508 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 276 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.389732 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 276 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:31.390297 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 276 candidate_status { last_received { term: 275 index: 180 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.390504 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 276 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.390923 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 276 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 276 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.390887 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 276 candidate_status { last_received { term: 275 index: 180 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:31.391088 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 276 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.391486 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 276 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 276 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.391583 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 276 election: Election decided. Result: candidate lost.
I1014 19:26:31.392082 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:31.392165 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Leader election lost for term 276. Reason: could not achieve majority
I1014 19:26:31.396275 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 275 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.396560 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.396929 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 275. Current term is 276. Ops: []
I1014 19:26:31.397444 10950 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 275.180, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I1014 19:26:31.397697 10950 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 LEADER]: Stepping down as leader of term 275
I1014 19:26:31.397748 10950 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:31.397956 10950 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 275.180, Last appended by leader: 180, Current term: 275, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.398172 10950 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:31.398242 10950 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:31.398412 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 275 FOLLOWER]: Advancing to term 276
I1014 19:26:31.400792 10946 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:31.400872 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:31.400956 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 275 FOLLOWER]: Advancing to term 276
I1014 19:26:31.402926 10946 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.403427 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 276 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:31.403659 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 276 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:31.404214 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 276 candidate_status { last_received { term: 275 index: 180 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:31.404412 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 276: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:31.404737 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 276 candidate_status { last_received { term: 275 index: 180 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:31.404922 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (vote granted)
I1014 19:26:31.407074 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 276 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 276.
I1014 19:26:31.407469 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 276 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:31.407554 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 276 election: Election decided. Result: candidate won.
I1014 19:26:31.407730 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 276 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 276: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:31.407888 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:31.407958 10946 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 FOLLOWER]: Leader election won for term 276
I1014 19:26:31.408068 10946 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:31.408288 10946 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 275.180, Last appended by leader: 180, Current term: 276, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.409248 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.409473 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.409678  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:31.411825  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:31.495968 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 276 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 275 index: 180. Preceding OpId from leader: term: 276 index: 181. (index mismatch)
I1014 19:26:31.496630 10946 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I1014 19:26:31.498136 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.498395 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.499820 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.500047 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.500903 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 275 index: 180. Preceding OpId from leader: term: 276 index: 181. (index mismatch)
I1014 19:26:31.501907 11291 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I1014 19:26:31.502373 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.502601 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.504578 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.504832 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.504971 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.505177 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.583688 11291 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 276 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.584033 11291 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.602722 10950 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 276 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:31.602838 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:31.602921 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 276 FOLLOWER]: Advancing to term 277
I1014 19:26:31.605213 10950 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.605787 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 277 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.605965 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 277 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:31.606632 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 277 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.606623 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 277 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:31.606827 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 277 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.606858 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 277 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.607225 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 277 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 277 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.607408 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 277 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 277 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.607501 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 277 election: Election decided. Result: candidate lost.
I1014 19:26:31.607832 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:31.607906 10950 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Leader election lost for term 277. Reason: could not achieve majority
I1014 19:26:31.622859 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 276. Current term is 277. Ops: []
I1014 19:26:31.623915 11627 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 276.181, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I1014 19:26:31.624234 10946 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 LEADER]: Stepping down as leader of term 276
I1014 19:26:31.624311 10946 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:31.624534 10946 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 276.181, Last appended by leader: 181, Current term: 276, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.624754 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:31.624809 10946 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:31.624965 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 276 FOLLOWER]: Advancing to term 277
I1014 19:26:31.686087 11128 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:31.686184 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:31.686264 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 276 FOLLOWER]: Advancing to term 277
I1014 19:26:31.688583 11128 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.689142 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 277 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.690068 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 277 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.690259 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
I1014 19:26:31.692298 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 277 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 277.
I1014 19:26:31.692711 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 277 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.692792 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 277 election: Election decided. Result: candidate won.
I1014 19:26:31.692898 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 277 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:31.693037 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:31.693163 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 FOLLOWER]: Leader election won for term 277
I1014 19:26:31.693586 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 277 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:31.693817 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 277: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:31.694106 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:31.694283 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 277 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 277: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:31.694384 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 276.181, Last appended by leader: 181, Current term: 277, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.695379 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 277 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.695701 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.695971  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:31.697890  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:31.742923 10950 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:31.743041 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:31.743120 10950 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 277 FOLLOWER]: Advancing to term 278
I1014 19:26:31.745409 10950 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.745975 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 278 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.746189 10950 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 278 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:31.746858 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 278 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:31.747027 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 277 FOLLOWER]: Advancing to term 278
I1014 19:26:31.747095 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (vote granted)
I1014 19:26:31.747078 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 278 candidate_status { last_received { term: 276 index: 181 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:31.747278 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 278 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.747675 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 278 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 278 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:31.749222 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 278.
I1014 19:26:31.749593 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 278 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:31.749667 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 278 election: Election decided. Result: candidate won.
I1014 19:26:31.749899 10950 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:31.749979 10950 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 FOLLOWER]: Leader election won for term 278
I1014 19:26:31.750102 10950 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:31.750313 10950 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 276.181, Last appended by leader: 181, Current term: 278, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.751291 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.751528 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.779649 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 277. Current term is 278. Ops: []
I1014 19:26:31.780190 11128 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I1014 19:26:31.780468 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 LEADER]: Stepping down as leader of term 277
I1014 19:26:31.780566 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:31.780791 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 277.182, Last appended by leader: 182, Current term: 277, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:31.781039 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:31.781095 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:31.781222 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 277 FOLLOWER]: Advancing to term 278
I1014 19:26:31.828977 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 276 index: 181. Preceding OpId from leader: term: 278 index: 182. (index mismatch)
I1014 19:26:31.829699 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I1014 19:26:31.831128 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.831348 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.832561 10950 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.832770 10950 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:31.833736 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 278 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 277 index: 182. Preceding OpId from leader: term: 278 index: 182. (term mismatch)
I1014 19:26:31.833833 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 181
I1014 19:26:31.833871 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 277.182
I1014 19:26:31.833936 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 278 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:31.834623 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.834830 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.835427 10950 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I1014 19:26:31.836796 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.836799 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.837064 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.837098 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:31.911855 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 278 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:31.912132 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:31.928443 11048 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.951s	user 0.001s	sys 0.000s
I1014 19:26:31.928501 11048 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:31.929443 11048 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:31.941159 11048 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:31.943189 11048 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:32.036340 11128 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 278 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:32.036487 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:32.036594 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 278 FOLLOWER]: Advancing to term 279
I1014 19:26:32.038403  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:32.039566  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:32.041317 11128 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.043417 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 279 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.044272 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 279 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.044503 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 279 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.043857 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 279 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.044940 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 279 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 279 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.045723 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 279 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.045924 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 279 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.046247  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 277 vs 279: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 279 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.046433 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 279 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 279 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.046519 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 279 election: Election decided. Result: candidate lost.
I1014 19:26:32.046768 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:32.046846 11128 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Leader election lost for term 279. Reason: could not achieve majority
I1014 19:26:32.049018 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 278. Current term is 279. Ops: []
I1014 19:26:32.049504 11419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 278.183, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I1014 19:26:32.049787 11419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 LEADER]: Stepping down as leader of term 278
I1014 19:26:32.049840 11419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:32.050038 11419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 278.183, Last appended by leader: 183, Current term: 278, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.050253 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:32.050323 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:32.050462 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 278 FOLLOWER]: Advancing to term 279
I1014 19:26:32.121866 10946 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:32.121973 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:32.122042 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 278 FOLLOWER]: Advancing to term 279
I1014 19:26:32.124104 10946 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.124677 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 279 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.125375 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 279 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.125474 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 279 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:32.125721 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 279: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:32.126075 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 279 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.126243 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:32.126241 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 279 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 279: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:32.128482 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 279 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 279.
I1014 19:26:32.128865 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 279 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.128938 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 279 election: Election decided. Result: candidate won.
I1014 19:26:32.129298 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:32.129389 10946 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 FOLLOWER]: Leader election won for term 279
I1014 19:26:32.136688 10946 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:32.137024 10946 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 278.183, Last appended by leader: 183, Current term: 279, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.138165 11862 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 279 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.139009 11862 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.168423 11128 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:32.168546 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:32.168617 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 279 FOLLOWER]: Advancing to term 280
I1014 19:26:32.170195 11128 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.170655 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 280 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.170881 11128 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 280 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.171404 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 280 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.171432 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 280 candidate_status { last_received { term: 278 index: 183 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.171641 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 280 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.171641 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 279 FOLLOWER]: Advancing to term 280
I1014 19:26:32.171725 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:32.172061 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 280 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 280 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.173528 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 280.
I1014 19:26:32.174021 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 280 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.174100 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 280 election: Election decided. Result: candidate won.
I1014 19:26:32.174456 11128 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:32.174556 11128 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 FOLLOWER]: Leader election won for term 280
I1014 19:26:32.174685 11128 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:32.174896 11128 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 278.183, Last appended by leader: 183, Current term: 280, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.176084 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.176340 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.176573  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:32.178369  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:32.221971 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 279. Current term is 280. Ops: []
I1014 19:26:32.222651 11862 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I1014 19:26:32.222913 11862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 LEADER]: Stepping down as leader of term 279
I1014 19:26:32.222986 11862 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:32.223184 11862 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 279.184, Last appended by leader: 184, Current term: 279, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.223417 11862 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:32.223502 11862 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:32.223644 11862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 279 FOLLOWER]: Advancing to term 280
I1014 19:26:32.277451 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 280 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 279 index: 184. Preceding OpId from leader: term: 280 index: 184. (term mismatch)
I1014 19:26:32.277530 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 183
I1014 19:26:32.277566 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 279.184
I1014 19:26:32.277631 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 280 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:32.278266 11128 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I1014 19:26:32.279718 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.279935 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.281356 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.281577 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.282265 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 278 index: 183. Preceding OpId from leader: term: 280 index: 184. (index mismatch)
I1014 19:26:32.282831 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I1014 19:26:32.284713 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.284716 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.285130 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.285259 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.286499 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.286732 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.370220 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 280 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.370468 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.387295 10946 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 280 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:32.387457 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:32.387531 10946 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 280 FOLLOWER]: Advancing to term 281
I1014 19:26:32.389797 10946 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.390275 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 281 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.390542 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 280. Current term is 281. Ops: []
I1014 19:26:32.391093 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 281 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:32.391278 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 281 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.391651 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 281 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 281 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.391894 11128 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 280.184, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I1014 19:26:32.392132 11128 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 LEADER]: Stepping down as leader of term 280
I1014 19:26:32.392200 11128 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:32.392396 11128 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 280.184, Last appended by leader: 184, Current term: 280, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.392635 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:32.392697 11128 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:32.392846 11128 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 280 FOLLOWER]: Advancing to term 281
I1014 19:26:32.393688 10946 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 281 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.394318 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 281 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.394572 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 281 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.395061 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 281 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 281 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.395164 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 281 election: Election decided. Result: candidate lost.
I1014 19:26:32.395454 11862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:32.395545 11862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Leader election lost for term 281. Reason: could not achieve majority
I1014 19:26:32.472148 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:32.472268 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:32.472354 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 280 FOLLOWER]: Advancing to term 281
I1014 19:26:32.474802 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.475337 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 281 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.475919 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 281 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.480100 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 281 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:32.480336 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:32.483330 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 281 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 281.
I1014 19:26:32.483669 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 281 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.483749 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 281 election: Election decided. Result: candidate won.
I1014 19:26:32.483759 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 281 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.483953 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 281: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:32.484064 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (election complete)
I1014 19:26:32.484127 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 FOLLOWER]: Leader election won for term 281
I1014 19:26:32.484222 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:32.484458 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 280.184, Last appended by leader: 184, Current term: 281, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.485543 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 281 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.485797 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.486286 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 281 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 281: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:32.500442 11862 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:32.500568 11862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.175s (starting election)
I1014 19:26:32.500634 11862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 281 FOLLOWER]: Advancing to term 282
I1014 19:26:32.502992 11862 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.503502 11862 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 282 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.504325 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 282 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:32.504555 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 281 FOLLOWER]: Advancing to term 282
I1014 19:26:32.504668 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:32.506853 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 282 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 282.
I1014 19:26:32.507320 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 282 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.507385 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 282 election: Election decided. Result: candidate won.
I1014 19:26:32.507604 10946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.179s (election complete)
I1014 19:26:32.507810 10946 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 FOLLOWER]: Leader election won for term 282
I1014 19:26:32.507809 11862 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 282 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.508488 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 282 candidate_status { last_received { term: 280 index: 184 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.508689 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 282 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.509078 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 282 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 282 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.514513 10946 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:32.516661 10946 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 280.184, Last appended by leader: 184, Current term: 282, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.517740 11862 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.517984 11862 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.570154 11048 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:32.571265 11048 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:32.571772 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 282 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 281. Current term is 282. Ops: []
I1014 19:26:32.572297 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I1014 19:26:32.572600 11575 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 LEADER]: Stepping down as leader of term 281
I1014 19:26:32.572681 11575 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:32.572968 11575 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 281.185, Last appended by leader: 185, Current term: 281, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.573176 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:32.573233 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:32.573341 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 281 FOLLOWER]: Advancing to term 282
I1014 19:26:32.576550 11048 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 278 vs 282: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.576869 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.577113 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.577350 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.577602 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.577849 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:32.578089 11048 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 282 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.612509 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 281 index: 185. Preceding OpId from leader: term: 282 index: 185. (term mismatch)
I1014 19:26:32.612598 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 184
I1014 19:26:32.612637 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 281.185
I1014 19:26:32.612702 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:32.613201 11862 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I1014 19:26:32.614529 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.614929 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.615767 11862 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.616076 11862 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.616403 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 282 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 280 index: 184. Preceding OpId from leader: term: 282 index: 185. (index mismatch)
I1014 19:26:32.616730 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.616960 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.617347 11862 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I1014 19:26:32.618674 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.618886 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.619143 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:32.619235 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:32.634860  9330 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.223s	user 0.002s	sys 0.000s
I1014 19:26:32.634944  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:32.635870  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:32.638200 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 282 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.638499 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.644171  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:32.646072  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:32.842185 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 282 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:32.842293 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:32.842378 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 282 FOLLOWER]: Advancing to term 283
I1014 19:26:32.844600 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.845144 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 283 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.845340 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 283 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.846017 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 283 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.846235 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 283 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.846221 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 283 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.846367 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 283 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.846669 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 283 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 283 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.846841 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 283 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 283 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.846920 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 283 election: Election decided. Result: candidate lost.
I1014 19:26:32.847141 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:32.847218 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Leader election lost for term 283. Reason: could not achieve majority
I1014 19:26:32.853118 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 282. Current term is 283. Ops: []
I1014 19:26:32.853706 10946 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 282.186, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I1014 19:26:32.853976 12032 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 LEADER]: Stepping down as leader of term 282
I1014 19:26:32.854041 12032 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:32.854249 12032 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 282.186, Last appended by leader: 186, Current term: 282, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.854454 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:32.854542 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:32.854661 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 282 FOLLOWER]: Advancing to term 283
I1014 19:26:32.915822 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:32.915928 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:32.916029 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 282 FOLLOWER]: Advancing to term 283
I1014 19:26:32.918301 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.918871 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 283 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.919100 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 283 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:32.919752 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 283 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:32.919970 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 283: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:32.919898 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 283 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.920127 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:32.920399 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 283 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 283: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:32.922731 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 283 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 283.
I1014 19:26:32.923794 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 283 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.923869 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 283 election: Election decided. Result: candidate won.
I1014 19:26:32.924226 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:32.924296 11575 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 FOLLOWER]: Leader election won for term 283
I1014 19:26:32.924450 11575 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:32.924662 11575 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 282.186, Last appended by leader: 186, Current term: 283, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.925712 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 283 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.926005 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:32.926231 11048 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:32.928598 11048 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:32.961266 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:32.961355 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:32.961458 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 283 FOLLOWER]: Advancing to term 284
I1014 19:26:32.963317 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.963876 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 284 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.964102 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 284 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:32.964650 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 284 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:32.964722 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 284 candidate_status { last_received { term: 282 index: 186 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:32.964869 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 283 FOLLOWER]: Advancing to term 284
I1014 19:26:32.964932 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 284 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.964951 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:32.965369 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 284 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 284 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:32.967424 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 284.
I1014 19:26:32.967836 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 284 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:32.967917 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 284 election: Election decided. Result: candidate won.
I1014 19:26:32.968320 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:32.968459 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 FOLLOWER]: Leader election won for term 284
I1014 19:26:32.968650 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:32.968917 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 282.186, Last appended by leader: 186, Current term: 284, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:32.970062 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:32.970317 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.009634 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 283. Current term is 284. Ops: []
I1014 19:26:33.010205 11419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I1014 19:26:33.010546 11575 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 LEADER]: Stepping down as leader of term 283
I1014 19:26:33.010607 11575 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:33.010862 11575 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 186, Committed index: 186, Last appended: 283.187, Last appended by leader: 187, Current term: 283, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.011073 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:33.011145 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:33.011277 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 283 FOLLOWER]: Advancing to term 284
I1014 19:26:33.048343 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 282 index: 186. Preceding OpId from leader: term: 284 index: 187. (index mismatch)
I1014 19:26:33.049111 11128 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I1014 19:26:33.050827 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.051035 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.051993 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.052162 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.052656 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 284 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 283 index: 187. Preceding OpId from leader: term: 284 index: 187. (term mismatch)
I1014 19:26:33.052733 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 186
I1014 19:26:33.052769 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 283.187
I1014 19:26:33.052839 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 284 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:33.052896 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.053077 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.054656 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 187, Last known committed idx: 186, Time since last communication: 0.000s
I1014 19:26:33.056258 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.056255 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.056658 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.056550 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.135265 11128 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 284 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.135543 11128 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.237715 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 284 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:33.237823 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:33.237907 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 284 FOLLOWER]: Advancing to term 285
I1014 19:26:33.240288 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.240890 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 285 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:33.241127 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 285 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.241720 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 285 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:33.241947 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 285 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.242146 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 285 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.242311 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 285 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.242578 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 284. Current term is 285. Ops: []
I1014 19:26:33.242765 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 285 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 285 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.242974 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 285 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 285 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.243041 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 285 election: Election decided. Result: candidate lost.
I1014 19:26:33.243309 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:33.243402 11575 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Leader election lost for term 285. Reason: could not achieve majority
I1014 19:26:33.243448 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 284.187, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.243716 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 LEADER]: Stepping down as leader of term 284
I1014 19:26:33.243767 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:33.243968 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 284.187, Last appended by leader: 187, Current term: 284, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.244160 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:33.244225 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:33.244349 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 284 FOLLOWER]: Advancing to term 285
I1014 19:26:33.252691 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:33.252789 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:33.252871 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 284 FOLLOWER]: Advancing to term 285
I1014 19:26:33.255125 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.255658 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 285 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.255861 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 285 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:33.256546 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 285 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.256593 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 285 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:33.256712 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:33.256794 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 285: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:33.257227 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 285 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 285: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:33.258852 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 285 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 285.
I1014 19:26:33.259685 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 285 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.259747 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 285 election: Election decided. Result: candidate won.
I1014 19:26:33.259980 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:33.260082 12032 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 FOLLOWER]: Leader election won for term 285
I1014 19:26:33.260253 12032 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:33.260524 12032 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 284.187, Last appended by leader: 187, Current term: 285, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.261699 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 285 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.261983 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.350934 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:33.351025 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:33.351111 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 285 FOLLOWER]: Advancing to term 286
I1014 19:26:33.353564 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.354055 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 286 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:33.354265 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 286 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.354832 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 286 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.355000 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 285 FOLLOWER]: Advancing to term 286
I1014 19:26:33.354938 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 286 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:33.355072 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:33.355172 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 286 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.355540 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 285. Current term is 286. Ops: []
I1014 19:26:33.356107 12032 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.356310 12032 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 LEADER]: Stepping down as leader of term 285
I1014 19:26:33.356365 12032 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:33.356566 12032 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 285.188, Last appended by leader: 188, Current term: 285, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.356767 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:33.356817 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:33.356930 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 FOLLOWER]: Advancing to term 286
I1014 19:26:33.357302 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 286 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 286.
I1014 19:26:33.358175 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 286 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.358269 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 286 election: Election decided. Result: candidate won.
I1014 19:26:33.358537 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 286 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 285 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 286 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.359148 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.198s (election complete)
I1014 19:26:33.359253 11575 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 FOLLOWER]: Leader election won for term 286
I1014 19:26:33.359709 11575 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:33.359942 11575 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 284.187, Last appended by leader: 187, Current term: 286, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.361490 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 286 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.361699 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.419914  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:33.420871  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:33.421929  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 282 vs 286: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.422343  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.422663  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.422916  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.423166  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.423609  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:33.423910  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 286 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.449185 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 286 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:33.449292 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:33.449365 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 286 FOLLOWER]: Advancing to term 287
I1014 19:26:33.451129 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.451634 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 287 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.451853 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 287 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:33.452383 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 287 candidate_status { last_received { term: 285 index: 188 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.452488 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 287 candidate_status { last_received { term: 285 index: 188 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:33.452592 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 286 FOLLOWER]: Advancing to term 287
I1014 19:26:33.452672 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (vote granted)
I1014 19:26:33.452721 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 287 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.453153 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 287 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 287 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.454864 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 287 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 287.
I1014 19:26:33.455241 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 287 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.455302 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 287 election: Election decided. Result: candidate won.
I1014 19:26:33.455658 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:33.455740 12032 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 FOLLOWER]: Leader election won for term 287
I1014 19:26:33.455875 12032 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:33.456076 12032 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 285.188, Last appended by leader: 188, Current term: 287, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.457206 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 287 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.457410 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.457621  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:33.459061  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:33.472067 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 286. Current term is 287. Ops: []
I1014 19:26:33.472674 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.472885 11575 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 287.189 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 187
I1014 19:26:33.472987 11575 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 LEADER]: Stepping down as leader of term 286
I1014 19:26:33.473033 11575 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:33.473217 11575 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 286.188, Last appended by leader: 188, Current term: 286, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.473407 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:33.473493 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:33.473610 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 286 FOLLOWER]: Advancing to term 287
I1014 19:26:33.560132 12180 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 287 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:33.560240 12180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:33.560329 12180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 287 FOLLOWER]: Advancing to term 288
I1014 19:26:33.562772 12180 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 288 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.563364 12180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 288 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:33.563774 12180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 288 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:33.564152 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 288 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:33.564357 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 288 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.564466 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 288 candidate_status { last_received { term: 284 index: 187 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:33.564666 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 287 FOLLOWER]: Advancing to term 288
I1014 19:26:33.565004 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 288 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 288 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.566596 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 288 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 288 because replica has last-logged OpId of term: 286 index: 188, which is greater than that of the candidate, which has last-logged OpId of term: 284 index: 187.
I1014 19:26:33.566716 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 288 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:33.566835 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:33.566910 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 288 FOLLOWER]: Advancing to term 289
I1014 19:26:33.567013 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 288 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 288 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 288 because replica has last-logged OpId of term: 286 index: 188, which is greater than that of the candidate, which has last-logged OpId of term: 284 index: 187.
I1014 19:26:33.567101 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 288 election: Election decided. Result: candidate lost.
I1014 19:26:33.567346 12180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:33.567432 12180 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 288 FOLLOWER]: Leader election lost for term 288. Reason: could not achieve majority
I1014 19:26:33.568428 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.568928 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 289 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:33.569126 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 289 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.569416 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 287. Current term is 289. Ops: []
I1014 19:26:33.569964 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 289 candidate_status { last_received { term: 286 index: 188 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:33.570053 10946 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 189, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.570263 10946 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 286.188 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 187
I1014 19:26:33.570209 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 289 candidate_status { last_received { term: 286 index: 188 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.570360 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 288 FOLLOWER]: Advancing to term 289
I1014 19:26:33.570456 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:33.570482 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 289 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.570595 12032 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Stepping down as leader of term 287
I1014 19:26:33.570690 12032 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:33.570861 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 289 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 289 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.570876 12032 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 287.189, Last appended by leader: 189, Current term: 287, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.571094 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:33.571135 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:33.571224 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 287 FOLLOWER]: Advancing to term 289
I1014 19:26:33.572502 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 289.
I1014 19:26:33.572823 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 289 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.572880 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 289 election: Election decided. Result: candidate won.
I1014 19:26:33.573107 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:33.573195 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 FOLLOWER]: Leader election won for term 289
I1014 19:26:33.573304 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:33.573549 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 286.188, Last appended by leader: 188, Current term: 289, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.574446 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.574685 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.592593 11048 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:33.593622 11048 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:33.596691 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 284 index: 187. Preceding OpId from leader: term: 289 index: 190. (index mismatch)
I1014 19:26:33.596740 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 289 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 287 index: 189. Preceding OpId from leader: term: 289 index: 190. (index mismatch)
I1014 19:26:33.597286 11419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.597487 11419 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 287.189 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 187
I1014 19:26:33.597640 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 187, Time since last communication: 0.000s
I1014 19:26:33.598444 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 187
I1014 19:26:33.598495 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 285.188
I1014 19:26:33.598569 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 289 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:33.598635 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 287.189
I1014 19:26:33.598670 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 289 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:33.599331 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.599710 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.600147 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.600358 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.605530 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.605733 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.606091 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.606312 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.607373 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.607662 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.607856 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.608081 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:33.608327 11048 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:33.610954 10946 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.611250 10946 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.611332 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:33.611572 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:33.611665 11048 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:33.705006 11048 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:33.705915 11048 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:33.984758 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 289 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:33.984905 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:33.985007 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 289 FOLLOWER]: Advancing to term 290
I1014 19:26:33.987627 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:33.988189 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 290 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:33.988802 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 290 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:33.989059 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 290 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.989802 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 290 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 290 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.989991 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 290 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:33.990701 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 290 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:33.990880 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 290 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.991228 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 290 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 290 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:33.991303 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 290 election: Election decided. Result: candidate lost.
I1014 19:26:33.991534 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:33.991627 12032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Leader election lost for term 290. Reason: could not achieve majority
I1014 19:26:34.010394 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 289. Current term is 290. Ops: []
I1014 19:26:34.011220 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 289.191, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I1014 19:26:34.011495 11419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 LEADER]: Stepping down as leader of term 289
I1014 19:26:34.011567 11419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:34.011776 11419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 289.191, Last appended by leader: 191, Current term: 289, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.012015 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:34.012080 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:34.012235 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 289 FOLLOWER]: Advancing to term 290
W1014 19:26:34.085027  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.907s	user 0.002s	sys 0.000s
I1014 19:26:34.085106  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:34.085860  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:34.086808  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 280 vs 289: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:34.087071  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:34.087344  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:34.087546  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:34.087740  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:34.087920  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 289 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.089886 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:34.089957 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:34.090050 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 289 FOLLOWER]: Advancing to term 290
I1014 19:26:34.092090 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.092654 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 290 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.092880 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 290 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.093425 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 290 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.093703 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 290: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:34.093741 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 290 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:34.093917 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (vote granted)
I1014 19:26:34.094158 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 290 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 290: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:34.096251 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 290 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 290.
I1014 19:26:34.096763 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 290 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.096884 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 290 election: Election decided. Result: candidate won.
I1014 19:26:34.097256 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:34.097335 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 FOLLOWER]: Leader election won for term 290
I1014 19:26:34.097476 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:34.097669 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 289.191, Last appended by leader: 191, Current term: 290, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.098630 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 290 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.098815 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.099089  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:34.101035  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:34.141968 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:34.142091 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.165s (starting election)
I1014 19:26:34.142154 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 290 FOLLOWER]: Advancing to term 291
I1014 19:26:34.144578 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.145125 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 291 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.145349 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 291 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.145759 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 291 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:34.145956 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 291 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.145929 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 291 candidate_status { last_received { term: 289 index: 191 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:34.146044 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 290 FOLLOWER]: Advancing to term 291
I1014 19:26:34.146122 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:34.146332 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 291 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 291 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.147903 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 291.
I1014 19:26:34.148216 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 291 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.148277 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 291 election: Election decided. Result: candidate won.
I1014 19:26:34.148504 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:34.148588 12032 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 FOLLOWER]: Leader election won for term 291
I1014 19:26:34.148916 12032 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:34.149219 12032 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 289.191, Last appended by leader: 191, Current term: 291, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.150269 12353 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.150599 12353 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.178107 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 290. Current term is 291. Ops: []
I1014 19:26:34.178701 12180 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I1014 19:26:34.178939 12180 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 LEADER]: Stepping down as leader of term 290
I1014 19:26:34.179003 12180 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:34.179250 12180 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 290.192, Last appended by leader: 192, Current term: 290, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.179428 12180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:34.179487 12180 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:34.179616 12180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 290 FOLLOWER]: Advancing to term 291
I1014 19:26:34.228452 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 289 index: 191. Preceding OpId from leader: term: 291 index: 192. (index mismatch)
I1014 19:26:34.229104 12353 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I1014 19:26:34.230572 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.230808 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.231883 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.232192 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.232722 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 291 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 290 index: 192. Preceding OpId from leader: term: 291 index: 192. (term mismatch)
I1014 19:26:34.232782 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 191
I1014 19:26:34.232825 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 290.192
I1014 19:26:34.232880 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 291 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:34.233427 12353 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I1014 19:26:34.234206 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.234452 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.235342 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.235690 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.235342 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.235844 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.334794 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 291 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:34.334892 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:34.334977 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 291 FOLLOWER]: Advancing to term 292
I1014 19:26:34.336573 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.336974 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 292 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.337139 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 292 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.337396 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 291 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.337882 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.337890 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 292 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:34.338104 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 292 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.338469 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 292 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.338543 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 291. Current term is 292. Ops: []
I1014 19:26:34.338634 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 292 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.339002 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 292 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 292 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.339033 12355 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 291.192, Next index: 193, Last known committed idx: 192, Time since last communication: 0.000s
I1014 19:26:34.339128 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 292 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 292 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.339223 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 292 election: Election decided. Result: candidate lost.
I1014 19:26:34.339278 12032 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 LEADER]: Stepping down as leader of term 291
I1014 19:26:34.339340 12032 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:34.339463 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:34.339529 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Leader election lost for term 292. Reason: could not achieve majority
I1014 19:26:34.339534 12032 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 192, Committed index: 192, Last appended: 291.192, Last appended by leader: 192, Current term: 291, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.339782 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:34.339833 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:34.339962 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 291 FOLLOWER]: Advancing to term 292
I1014 19:26:34.426277 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:34.426383 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:34.426457 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 291 FOLLOWER]: Advancing to term 292
I1014 19:26:34.428849 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.429356 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 292 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.430713 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 292 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.431170 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 292 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.431385 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:34.431742 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 292 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:34.431957 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 292: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:34.432382 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 292 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 292: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:34.433352 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 292 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 292.
I1014 19:26:34.433691 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 292 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.433774 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 292 election: Election decided. Result: candidate won.
I1014 19:26:34.434013 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:34.434101 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 FOLLOWER]: Leader election won for term 292
I1014 19:26:34.434245 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:34.434428 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 192, Committed index: 192, Last appended: 291.192, Last appended by leader: 192, Current term: 292, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.435472 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 292 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.435737 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.437167 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:34.439301 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:34.486901 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:34.486999 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:34.487079 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 292 FOLLOWER]: Advancing to term 293
I1014 19:26:34.489457 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.490039 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 293 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.490730 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 293 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.490804 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 293 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.490972 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 292 FOLLOWER]: Advancing to term 293
I1014 19:26:34.491070 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:34.491411 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 293 candidate_status { last_received { term: 291 index: 192 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:34.491639 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 293 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.492216 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 293 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 293 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.493322 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 293.
I1014 19:26:34.493721 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 293 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.493804 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 293 election: Election decided. Result: candidate won.
I1014 19:26:34.494143 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.211s (election complete)
I1014 19:26:34.494223 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 FOLLOWER]: Leader election won for term 293
I1014 19:26:34.494400 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:34.494678 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 192, Committed index: 192, Last appended: 291.192, Last appended by leader: 192, Current term: 293, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.495712 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.496054 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.548487 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 292. Current term is 293. Ops: []
I1014 19:26:34.549224 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 193, Last known committed idx: 192, Time since last communication: 0.000s
I1014 19:26:34.549515 11575 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 293.193 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 192
I1014 19:26:34.549619 11419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 LEADER]: Stepping down as leader of term 292
I1014 19:26:34.549693 11419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:34.549895 11419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 192, Committed index: 192, Last appended: 292.193, Last appended by leader: 193, Current term: 292, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.550098 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:34.550156 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:34.550294 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 292 FOLLOWER]: Advancing to term 293
I1014 19:26:34.595639 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 293 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 292 index: 193. Preceding OpId from leader: term: 293 index: 193. (term mismatch)
I1014 19:26:34.595713 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 192
I1014 19:26:34.595758 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 292.193
I1014 19:26:34.595811 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 293 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:34.596487 12180 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 193, Last known committed idx: 192, Time since last communication: 0.000s
I1014 19:26:34.599354 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.599617 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.600015 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 291 index: 192. Preceding OpId from leader: term: 293 index: 193. (index mismatch)
I1014 19:26:34.600409 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.600658 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.600594 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 193, Last known committed idx: 192, Time since last communication: 0.000s
I1014 19:26:34.600826 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.601083 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.601932 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.602262 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.602262 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.602543 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:34.678293 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 293 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.678542 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.793509 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 293 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:34.793613 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:34.793699 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 293 FOLLOWER]: Advancing to term 294
I1014 19:26:34.796124 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.799412 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 294 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.799789 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 294 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.800369 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 294 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.800595 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 294 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.800966 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 294 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 294 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.803956 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 294 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:34.804157 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 294 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.804720 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 294 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 294 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.804848 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 294 election: Election decided. Result: candidate lost.
I1014 19:26:34.804842 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 293. Current term is 294. Ops: []
I1014 19:26:34.805157 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:34.805263 11419 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Leader election lost for term 294. Reason: could not achieve majority
I1014 19:26:34.805675 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 293.193, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I1014 19:26:34.805930 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Stepping down as leader of term 293
I1014 19:26:34.805984 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:34.806182 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 293.193, Last appended by leader: 193, Current term: 293, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.806390 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:34.806452 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:34.806622 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 293 FOLLOWER]: Advancing to term 294
I1014 19:26:34.859993 12355 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:34.860116 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:34.860189 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 293 FOLLOWER]: Advancing to term 294
I1014 19:26:34.862612 12355 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.863198 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 294 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.863988 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 294 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:34.864189 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:34.866508 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 294 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 294.
I1014 19:26:34.863798 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 294 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:34.867043 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 294 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.867121 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 294 election: Election decided. Result: candidate won.
I1014 19:26:34.867348 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:34.867486 12355 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 FOLLOWER]: Leader election won for term 294
I1014 19:26:34.867449 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 294 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:34.867687 12355 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:34.867780 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 294: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:34.867964 12355 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 293.193, Last appended by leader: 193, Current term: 294, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.868271 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 294 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 294: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:34.869503 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 294 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.869727 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.907363 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:34.907464 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.184s (starting election)
I1014 19:26:34.907580 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 294 FOLLOWER]: Advancing to term 295
I1014 19:26:34.909919 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.910439 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 295 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:34.911368 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 295 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:34.911602 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 295 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.912724 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 295 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 295 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:34.913202 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 295 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.914490 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 295 candidate_status { last_received { term: 293 index: 193 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:34.914695 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 294 FOLLOWER]: Advancing to term 295
I1014 19:26:34.914788 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:34.917129 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 295 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 295.
I1014 19:26:34.917748 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 295 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:34.917857 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 295 election: Election decided. Result: candidate won.
I1014 19:26:34.918100 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:34.918203 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 FOLLOWER]: Leader election won for term 295
I1014 19:26:34.927589 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:34.927839 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 293.193, Last appended by leader: 193, Current term: 295, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.929292 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:34.929740 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:34.988026 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 294. Current term is 295. Ops: []
I1014 19:26:34.988603 12032 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I1014 19:26:34.988903 12032 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 295.194 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 193
I1014 19:26:34.989025 12355 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 LEADER]: Stepping down as leader of term 294
I1014 19:26:34.989114 12355 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:34.989328 12355 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 193, Committed index: 193, Last appended: 294.194, Last appended by leader: 194, Current term: 294, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:34.989567 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:34.989621 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:34.989755 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 294 FOLLOWER]: Advancing to term 295
I1014 19:26:35.025102 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 295 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 293 index: 193. Preceding OpId from leader: term: 295 index: 194. (index mismatch)
I1014 19:26:35.025853 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I1014 19:26:35.027348 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.027634 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.028604 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.028825 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.029223 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.029378 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.030040 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 295 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 294 index: 194. Preceding OpId from leader: term: 295 index: 194. (term mismatch)
I1014 19:26:35.030145 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 193
I1014 19:26:35.030210 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 294.194
I1014 19:26:35.030313 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 295 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:35.030861 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 193, Time since last communication: 0.000s
I1014 19:26:35.032485 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.032795 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.032999 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 295 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.033275 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.129393 12180 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 295 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:35.129521 12180 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:35.129609 12180 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 295 FOLLOWER]: Advancing to term 296
I1014 19:26:35.131988 12180 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.132592 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 295 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:35.132689 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:35.132751 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 295 FOLLOWER]: Advancing to term 296
I1014 19:26:35.134989 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 296 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.135267 12180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 296 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.135460 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 296 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.135530 12180 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 296 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.135661 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 296 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.136183 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 296 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:35.136288 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 296 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.136396 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 296 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 296: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:35.136454 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 296: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:35.136813 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 296 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 296: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:35.136987 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 296 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.137207 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 296 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.137651 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 296 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 296 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.137748 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 296 election: Election decided. Result: candidate lost.
I1014 19:26:35.137996 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:35.138146 12032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 296 FOLLOWER]: Leader election lost for term 296. Reason: could not achieve majority
I1014 19:26:35.138185 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 296 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 296 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 296: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:35.139176 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 296 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.139328 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 295. Current term is 296. Ops: []
I1014 19:26:35.139350 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 296 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.139747 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 296 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 296 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.139847 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 296 election: Election decided. Result: candidate lost.
I1014 19:26:35.139799 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 295.194, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I1014 19:26:35.140048 11419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Stepping down as leader of term 295
I1014 19:26:35.140112 11419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:35.140128 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:35.140228 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Leader election lost for term 296. Reason: could not achieve majority
I1014 19:26:35.140297 11419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 295.194, Last appended by leader: 194, Current term: 295, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.140517 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:35.140588 11419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:35.140727 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 295 FOLLOWER]: Advancing to term 296
I1014 19:26:35.242336 11419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 296 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:35.242466 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:35.242540 11419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 296 FOLLOWER]: Advancing to term 297
I1014 19:26:35.244449 11419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.244854 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 297 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.245060 11419 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 297 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.245527 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 297 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:35.245697 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 296 FOLLOWER]: Advancing to term 297
I1014 19:26:35.245767 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.200s (vote granted)
I1014 19:26:35.245862 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 297 candidate_status { last_received { term: 295 index: 194 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.246042 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 296 FOLLOWER]: Advancing to term 297
I1014 19:26:35.246126 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (vote granted)
I1014 19:26:35.247570 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 297 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 297.
I1014 19:26:35.247894 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 297.
I1014 19:26:35.247915 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 297 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.247973 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 297 election: Election decided. Result: candidate won.
I1014 19:26:35.248230 11419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:35.248289 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 297 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.248363 11419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 FOLLOWER]: Leader election won for term 297
I1014 19:26:35.248531 11419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:35.248744 11419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 295.194, Last appended by leader: 194, Current term: 297, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.249608 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.249783 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.340189 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 297 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 295 index: 194. Preceding OpId from leader: term: 297 index: 195. (index mismatch)
I1014 19:26:35.340685 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I1014 19:26:35.342047 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.343225 11419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.343502 11419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.344090 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 295 index: 194. Preceding OpId from leader: term: 297 index: 195. (index mismatch)
I1014 19:26:35.344368 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.344614 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.344588 11419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I1014 19:26:35.345919 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.346164 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.346060 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.346354 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.364135 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:35.394116  9330 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.935s	user 0.001s	sys 0.000s
I1014 19:26:35.394173  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:35.395117  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:35.396225  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 287 vs 297: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.396484  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.396708  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.396903  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.397107  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.441058 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 297 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.441334 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.444599 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 297 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:35.444734 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:35.444828 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 297 FOLLOWER]: Advancing to term 298
I1014 19:26:35.446936 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.447466 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 298 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.447717 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 298 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.448226 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 298 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:35.448348 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 298 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.448462 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 298 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.448554 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 298 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.448801 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 298 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 298 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.448915 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 298 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 298 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.448985 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 298 election: Election decided. Result: candidate lost.
I1014 19:26:35.449291 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:35.449374 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Leader election lost for term 298. Reason: could not achieve majority
I1014 19:26:35.462409 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 297. Current term is 298. Ops: []
I1014 19:26:35.463014 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 297.195, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I1014 19:26:35.463312 11575 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 LEADER]: Stepping down as leader of term 297
I1014 19:26:35.463392 11575 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:35.463681 11575 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 297.195, Last appended by leader: 195, Current term: 297, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.463929 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:35.463996 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:35.464162 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 297 FOLLOWER]: Advancing to term 298
W1014 19:26:35.480082  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.379s	user 0.002s	sys 0.000s
I1014 19:26:35.480126  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:35.481082  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:35.482147  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 290 vs 298: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 298 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.482440  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 298 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.482648  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 298 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.482851  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 298 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.541812 12355 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:35.541893 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:35.541940 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 297 FOLLOWER]: Advancing to term 298
I1014 19:26:35.544015 12355 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.544472 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 298 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.544618 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 298 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.545114 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 298 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.545328 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 298: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:35.545250 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 298 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.545423 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:35.545789 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 298 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 298: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:35.547037 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 298 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 298.
I1014 19:26:35.547338 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 298 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.547389 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 298 election: Election decided. Result: candidate won.
I1014 19:26:35.547648 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:35.547699 12355 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 FOLLOWER]: Leader election won for term 298
I1014 19:26:35.547786 12355 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:35.547942 12355 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 297.195, Last appended by leader: 195, Current term: 298, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.548804 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 298 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.549012 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.549170  9330 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:35.550456  9330 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:35.563912 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:35.563987 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:35.564040 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 298 FOLLOWER]: Advancing to term 299
I1014 19:26:35.565613 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.565992 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 299 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.566135 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 299 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.566725 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 299 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:35.566799 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 299 candidate_status { last_received { term: 297 index: 195 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.566898 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 299 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.566973 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 298 FOLLOWER]: Advancing to term 299
I1014 19:26:35.567045 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:35.567282 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 299 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 299 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.568622 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 299.
I1014 19:26:35.568898 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 299 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.568948 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 299 election: Election decided. Result: candidate won.
I1014 19:26:35.569103 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:35.569159 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 FOLLOWER]: Leader election won for term 299
I1014 19:26:35.569242 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:35.569399 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 297.195, Last appended by leader: 195, Current term: 299, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.570435 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.570603 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.570775  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:35.572522  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:35.649534 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 298. Current term is 299. Ops: []
I1014 19:26:35.650244 12032 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I1014 19:26:35.650478 12032 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 LEADER]: Stepping down as leader of term 298
I1014 19:26:35.650542 12032 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:35.650712 12032 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 298.196, Last appended by leader: 196, Current term: 298, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.650888 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:35.650928 12032 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:35.651027 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 298 FOLLOWER]: Advancing to term 299
I1014 19:26:35.664918 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 299 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 298 index: 196. Preceding OpId from leader: term: 299 index: 196. (term mismatch)
I1014 19:26:35.664975 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 195
I1014 19:26:35.664994 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 298.196
I1014 19:26:35.665026 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 299 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:35.665612 12180 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I1014 19:26:35.667053 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.667248 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.668285 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.668555 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.668988 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 297 index: 195. Preceding OpId from leader: term: 299 index: 196. (index mismatch)
I1014 19:26:35.669441 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.669606 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.669548 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I1014 19:26:35.671085 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.670960 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.671361 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.671365 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:35.758533 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.758739 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.861675 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 299 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:35.861760 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:35.861824 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 299 FOLLOWER]: Advancing to term 300
I1014 19:26:35.863951 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.864470 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 300 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.864636 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 300 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.865320 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 300 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.865545 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 300 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.865545 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 300 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.865950 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 300 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 300 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.869576 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 300 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.870000 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 300 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 300 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.870074 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 300 election: Election decided. Result: candidate lost.
I1014 19:26:35.870262 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:35.870327 12032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Leader election lost for term 300. Reason: could not achieve majority
I1014 19:26:35.874053 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 299. Current term is 300. Ops: []
I1014 19:26:35.874841 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 299.196, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I1014 19:26:35.875103 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 LEADER]: Stepping down as leader of term 299
I1014 19:26:35.875141 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:35.875313 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 299.196, Last appended by leader: 196, Current term: 299, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.875524 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:35.875563 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:35.875681 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 299 FOLLOWER]: Advancing to term 300
W1014 19:26:35.921340 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.482s	user 0.003s	sys 0.000s
I1014 19:26:35.921406 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:35.922544 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:35.923877 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 292 vs 299: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.924294 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.924638 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.924938 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.925251 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:35.925590 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 299 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.940084 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:35.940191 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:35.940284 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 299 FOLLOWER]: Advancing to term 300
I1014 19:26:35.942903 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.943377 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 300 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:35.943557 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 300 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.944249 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 300 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:35.944262 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 300 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.944419 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:35.944435 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 300: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:35.944811 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 300 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 300: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:35.946470 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 300 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 300.
I1014 19:26:35.946868 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 300 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.946961 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 300 election: Election decided. Result: candidate won.
I1014 19:26:35.947330 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:35.947677 12487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 FOLLOWER]: Leader election won for term 300
I1014 19:26:35.947882 12487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:35.949584 12487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 299.196, Last appended by leader: 196, Current term: 300, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.951889 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 300 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.952108 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:35.952302 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:35.954401 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:35.988407 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:35.988517 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.153s (starting election)
I1014 19:26:35.988575 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 300 FOLLOWER]: Advancing to term 301
I1014 19:26:35.989991 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.990371 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 301 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.990530 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 301 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:35.991080 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 301 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:35.991128 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 301 candidate_status { last_received { term: 299 index: 196 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:35.991231 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 300 FOLLOWER]: Advancing to term 301
I1014 19:26:35.991298 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (vote granted)
I1014 19:26:35.991328 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 301 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.991724 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 301 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 301 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:35.992803 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 301.
I1014 19:26:35.993096 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 301 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:35.993160 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 301 election: Election decided. Result: candidate won.
I1014 19:26:35.993340 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.222s (election complete)
I1014 19:26:35.993391 12032 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 FOLLOWER]: Leader election won for term 301
I1014 19:26:35.993466 12032 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:35.993613 12032 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 299.196, Last appended by leader: 196, Current term: 301, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:35.994335 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:35.994475 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.038305 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 300. Current term is 301. Ops: []
I1014 19:26:36.038880 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I1014 19:26:36.039126 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 LEADER]: Stepping down as leader of term 300
I1014 19:26:36.039180 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:36.039386 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 300.197, Last appended by leader: 197, Current term: 300, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.039583 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:36.039664 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:36.039786 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 300 FOLLOWER]: Advancing to term 301
I1014 19:26:36.096954 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 301 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 300 index: 197. Preceding OpId from leader: term: 301 index: 197. (term mismatch)
I1014 19:26:36.097033 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 196
I1014 19:26:36.097069 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 300.197
I1014 19:26:36.097133 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 301 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:36.097625 12355 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I1014 19:26:36.099030 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.099287 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.099802 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 299 index: 196. Preceding OpId from leader: term: 301 index: 197. (index mismatch)
I1014 19:26:36.099963 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.100157 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.101495 12355 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I1014 19:26:36.102998 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.103190 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.103305 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.103514 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.101209 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.114368 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.195420 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 301 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.195677 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.288702 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 301 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:36.288790 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:36.288857 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 301 FOLLOWER]: Advancing to term 302
I1014 19:26:36.290372 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.290804 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 302 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.290949 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 302 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:36.291494 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 302 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:36.291527 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 302 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:36.291641 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 302 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.291671 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 302 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.291945 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 302 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 302 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.292084 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 302 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 302 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.292135 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 302 election: Election decided. Result: candidate lost.
I1014 19:26:36.292289 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:36.292342 12487 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Leader election lost for term 302. Reason: could not achieve majority
I1014 19:26:36.310678 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 301. Current term is 302. Ops: []
I1014 19:26:36.311488 12355 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 301.197, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I1014 19:26:36.311725 12355 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 LEADER]: Stepping down as leader of term 301
I1014 19:26:36.311760 12355 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:36.311920 12355 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 301.197, Last appended by leader: 197, Current term: 301, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.312086 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:36.312119 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:36.312263 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 301 FOLLOWER]: Advancing to term 302
I1014 19:26:36.394143 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:36.394210 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:36.394258 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 301 FOLLOWER]: Advancing to term 302
I1014 19:26:36.395718 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.396119 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 302 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.396257 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 302 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:36.396759 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 302 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:36.396904 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (vote granted)
I1014 19:26:36.398921 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 302 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 302.
I1014 19:26:36.399196 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 302 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:36.399278 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 302 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.399332 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 302 election: Election decided. Result: candidate won.
I1014 19:26:36.399370 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 302: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:36.399525 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (election complete)
I1014 19:26:36.399582 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 FOLLOWER]: Leader election won for term 302
I1014 19:26:36.399659 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:36.399816 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 301.197, Last appended by leader: 197, Current term: 302, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.400529 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 302 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.400696 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.400951 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 302 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 302: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:36.432901 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:36.432981 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.191s (starting election)
I1014 19:26:36.433033 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 302 FOLLOWER]: Advancing to term 303
I1014 19:26:36.435196 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.435767 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 303 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.435946 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 303 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:36.436568 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 303 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:36.436723 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 302 FOLLOWER]: Advancing to term 303
I1014 19:26:36.436782 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:36.437835 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 303 candidate_status { last_received { term: 301 index: 197 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:36.438028 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 303 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.438453 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 303 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 303 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.438896 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 303.
I1014 19:26:36.439182 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 303 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.439234 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 303 election: Election decided. Result: candidate won.
I1014 19:26:36.439430 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:36.439524 12487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 FOLLOWER]: Leader election won for term 303
I1014 19:26:36.439654 12487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:36.439898 12487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 301.197, Last appended by leader: 197, Current term: 303, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.440899 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.441188 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.509296 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 302. Current term is 303. Ops: []
I1014 19:26:36.509862 12180 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I1014 19:26:36.510040 12180 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 303.198 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 197
I1014 19:26:36.510104 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 LEADER]: Stepping down as leader of term 302
I1014 19:26:36.510150 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:36.510344 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 302.198, Last appended by leader: 198, Current term: 302, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.510538 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:36.510573 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:36.510663 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 302 FOLLOWER]: Advancing to term 303
I1014 19:26:36.520555 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 303 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 302 index: 198. Preceding OpId from leader: term: 303 index: 198. (term mismatch)
I1014 19:26:36.520619 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 197
I1014 19:26:36.520668 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 302.198
I1014 19:26:36.520728 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 303 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:36.521270 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I1014 19:26:36.522843 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.523059 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.523703 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.523874 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.524844 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 301 index: 197. Preceding OpId from leader: term: 303 index: 198. (index mismatch)
I1014 19:26:36.525243 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.525415 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.525540 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I1014 19:26:36.526948 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.526967 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.527127 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.527143 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.621490 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 303 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.621717 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.709367 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 303 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:36.709476 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:36.709594 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 303 FOLLOWER]: Advancing to term 304
I1014 19:26:36.711606 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.712141 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 304 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.712399 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 304 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:36.712419 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:36.713081 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 304 candidate_status { last_received { term: 303 index: 198 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:36.713265 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 304 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.713346 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:36.713637 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 304 candidate_status { last_received { term: 303 index: 198 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:36.713835 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 304 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.714776 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 304 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 304 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.714987 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 304 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 304 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:36.715070 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 304 election: Election decided. Result: candidate lost.
I1014 19:26:36.715483 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:36.715590 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Leader election lost for term 304. Reason: could not achieve majority
I1014 19:26:36.715973 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 303. Current term is 304. Ops: []
I1014 19:26:36.717115 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 303.198, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I1014 19:26:36.717334 11575 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Stepping down as leader of term 303
I1014 19:26:36.717383 11575 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:36.717588 11575 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 303.199, Last appended by leader: 199, Current term: 303, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.717865 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:36.717945 11575 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:36.718056 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 303 FOLLOWER]: Advancing to term 304
I1014 19:26:36.817306 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:36.817394 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:36.817446 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 303 FOLLOWER]: Advancing to term 304
I1014 19:26:36.818044 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 304 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:36.818114 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:36.818156 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 304 FOLLOWER]: Advancing to term 305
I1014 19:26:36.819075 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 304 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.819547 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 304 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:36.819689 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 304 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:36.820186 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.820402 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 304 candidate_status { last_received { term: 303 index: 199 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:36.820587 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 304. Current term is 305.
I1014 19:26:36.820655 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 305 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.820801 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 305 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:26:36.820988 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 304 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 304. Current term is 305.
I1014 19:26:36.821020 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 304 election: Cancelling election due to peer responding with higher term
I1014 19:26:36.821375 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:36.821317 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 304 candidate_status { last_received { term: 303 index: 199 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:36.821429 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 304 FOLLOWER]: Advancing to term 305
I1014 19:26:36.821395 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 305 candidate_status { last_received { term: 303 index: 199 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:36.821544 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 304: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:36.822021 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 304 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 304: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:36.822060 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 305 candidate_status { last_received { term: 303 index: 199 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:36.822197 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 304 FOLLOWER]: Advancing to term 305
I1014 19:26:36.822286 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.217s (vote granted)
I1014 19:26:36.823148 12032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 305 FOLLOWER]: Leader election lost for term 304. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 304. Current term is 305.
I1014 19:26:36.823253 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.165s (vote granted)
I1014 19:26:36.824349 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 305.
I1014 19:26:36.824429 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 305 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 305.
I1014 19:26:36.824753 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 305 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:36.824808 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 305 election: Election decided. Result: candidate won.
I1014 19:26:36.825011 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 305 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:36.825471 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:36.825564 11575 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Leader election won for term 305
I1014 19:26:36.825678 11575 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:36.825909 11575 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 303.199, Last appended by leader: 199, Current term: 305, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:36.826939 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.827204 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.906245 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 305 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 303 index: 199. Preceding OpId from leader: term: 305 index: 200. (index mismatch)
I1014 19:26:36.906947 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 198, Time since last communication: 0.000s
I1014 19:26:36.908370 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.908665 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.922452 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:36.922646 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.922878 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.924203 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:36.924836 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 303 index: 198. Preceding OpId from leader: term: 305 index: 200. (index mismatch)
I1014 19:26:36.924813 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.925025 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:36.925390 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 198, Time since last communication: 0.000s
I1014 19:26:36.927295 12180 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.927654 12180 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:36.927994 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:36.928216 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.010356 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 305 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:37.010447 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:37.010501 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 305 FOLLOWER]: Advancing to term 306
I1014 19:26:37.012713 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.013202 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 306 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.013628 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 306 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.013975 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 306 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.014161 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 306 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.014201 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 306 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.014410 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 306 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.014690 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 306 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 306 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.014863 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 306 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 306 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.014931 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 306 election: Election decided. Result: candidate lost.
I1014 19:26:37.015103 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:37.015164 12355 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Leader election lost for term 306. Reason: could not achieve majority
I1014 19:26:37.015398 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 305. Current term is 306. Ops: []
I1014 19:26:37.019434 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 305.200, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I1014 19:26:37.019491 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 305 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.019712 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.019834 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 LEADER]: Stepping down as leader of term 305
I1014 19:26:37.019930 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.020164 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 305.200, Last appended by leader: 200, Current term: 305, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.020336 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:37.022213 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:37.022289 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 305 FOLLOWER]: Advancing to term 306
I1014 19:26:37.048756 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:37.049456 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:37.050282 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 305 vs 306: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.050595 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.050822 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.051002 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.051169 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.051347 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 306 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.123468 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:37.123577 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:37.123649 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 305 FOLLOWER]: Advancing to term 306
I1014 19:26:37.125804 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 306 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.126345 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 306 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.127769 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 306 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.127959 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 306: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:37.128391 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 306 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.128394 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 306 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 306: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:37.129006 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 306 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.129204 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:37.131345 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 306 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 306.
I1014 19:26:37.131958 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 306 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.132021 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 306 election: Election decided. Result: candidate won.
I1014 19:26:37.132987 12032 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.133045 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:37.133087 12032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 306 FOLLOWER]: Advancing to term 307
I1014 19:26:37.134827 12032 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.135268 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 307 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.135408 12032 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 307 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.136029 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 307 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.136181 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 306 FOLLOWER]: Advancing to term 307
I1014 19:26:37.136267 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:37.136458 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 307 candidate_status { last_received { term: 305 index: 200 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.136633 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 306 FOLLOWER]: Advancing to term 307
I1014 19:26:37.136723 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (vote granted)
I1014 19:26:37.138108 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 307 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 307.
I1014 19:26:37.138419 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 307 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.138432 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:37.138476 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 307 election: Election decided. Result: candidate won.
I1014 19:26:37.138519 11440 raft_consensus.cc:2512] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 307 FOLLOWER]: Leader election decision vote started in defunct term 306: won
I1014 19:26:37.138669 12032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.219s (election complete)
I1014 19:26:37.138723 12032 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 FOLLOWER]: Leader election won for term 307
I1014 19:26:37.138803 12032 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:37.138990 12032 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 305.200, Last appended by leader: 200, Current term: 307, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.140573 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 307.
I1014 19:26:37.140496 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.140676 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.140974 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 307 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
W1014 19:26:37.176236  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.604s	user 0.001s	sys 0.000s
I1014 19:26:37.176308  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:37.177176  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:37.178695  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 299 vs 307: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 307 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.179010  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 307 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.179270  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 307 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.179546  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 307 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.218669 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 305 index: 200. Preceding OpId from leader: term: 307 index: 201. (index mismatch)
I1014 19:26:37.219357 12355 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I1014 19:26:37.220268 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 307 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 305 index: 200. Preceding OpId from leader: term: 307 index: 201. (index mismatch)
I1014 19:26:37.220613 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.220834 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.221431 12355 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I1014 19:26:37.222280 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.222481 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.222631 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.222857 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.223589 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.223789 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.223856 12032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.224025 12032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.224953 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 307 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.225136 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.325573 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 307 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:37.325680 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:37.325764 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 307 FOLLOWER]: Advancing to term 308
I1014 19:26:37.327841 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.328366 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 308 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.328615 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 308 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.329282 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 308 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.329493 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 308 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.329489 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 308 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.329736 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 308 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.329936 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 308 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 308 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.330237 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 308 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 308 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.330307 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 308 election: Election decided. Result: candidate lost.
I1014 19:26:37.333768 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:37.333845 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Leader election lost for term 308. Reason: could not achieve majority
I1014 19:26:37.349114 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 307. Current term is 308. Ops: []
I1014 19:26:37.349599 12032 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 307.201, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.349834 12355 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 LEADER]: Stepping down as leader of term 307
I1014 19:26:37.349881 12355 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:37.350059 12355 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 307.201, Last appended by leader: 201, Current term: 307, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.350234 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:37.350272 12355 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:37.350386 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 307 FOLLOWER]: Advancing to term 308
W1014 19:26:37.361205  9330 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.811s	user 0.001s	sys 0.000s
I1014 19:26:37.361248  9330 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:37.362094  9330 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:37.363148  9330 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 298 vs 308: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.363399  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.363638  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.363837  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.364045  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.364254  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:37.364465  9330 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 308 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.416194 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:37.416309 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:26:37.416381 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 307 FOLLOWER]: Advancing to term 308
I1014 19:26:37.418622 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.419137 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 308 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.419370 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 308 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.419975 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 308 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.419993 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 308 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.420142 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (vote granted)
I1014 19:26:37.420181 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 308: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:37.422168 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 308 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 308.
I1014 19:26:37.422725 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 308 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.422788 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 308 election: Election decided. Result: candidate won.
I1014 19:26:37.423022 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 308 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 308: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:37.423308 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:26:37.423390 12487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 FOLLOWER]: Leader election won for term 308
I1014 19:26:37.423518 12487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.423758 12487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 307.201, Last appended by leader: 201, Current term: 308, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.424693 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 308 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.424885 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.425065 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:37.426599 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:37.452081 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.452170 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:37.452248 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 308 FOLLOWER]: Advancing to term 309
I1014 19:26:37.454326 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.454805 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 309 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.454996 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 309 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.455457 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 309 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.455619 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 308 FOLLOWER]: Advancing to term 309
I1014 19:26:37.455677 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:37.455605 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 309 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.455816 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 309 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.456262 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 309 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 309 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.457628 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 309 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 309.
I1014 19:26:37.458201 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 309 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.458276 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 309 election: Election decided. Result: candidate won.
I1014 19:26:37.458534 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.205s (election complete)
I1014 19:26:37.458597 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 FOLLOWER]: Leader election won for term 309
I1014 19:26:37.458905 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:37.459130 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 307.201, Last appended by leader: 201, Current term: 309, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.460177 13232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 309 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.460484 13232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.460680  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:37.462383  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:37.530360 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 309 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 308. Current term is 309. Ops: []
I1014 19:26:37.531009 11575 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.531262 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 LEADER]: Stepping down as leader of term 308
I1014 19:26:37.531332 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.531570 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 308.202, Last appended by leader: 202, Current term: 308, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.531741 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:37.531808 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:37.531930 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 308 FOLLOWER]: Advancing to term 309
I1014 19:26:37.554349 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 308 index: 202. Preceding OpId from leader: term: 309 index: 202. (term mismatch)
I1014 19:26:37.554431 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 201
I1014 19:26:37.554474 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 308.202
I1014 19:26:37.554570 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:37.555150 13232 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.556449 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 309 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.556665 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.557880 13232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 309 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.558100 13232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.558385 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 309 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 307 index: 201. Preceding OpId from leader: term: 309 index: 202. (index mismatch)
I1014 19:26:37.559208 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.559666 12355 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 309 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.559747 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:26:37.559803 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 309 FOLLOWER]: Advancing to term 310
I1014 19:26:37.559727 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 309 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.559927 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.561985 12355 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.562500 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 310 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.563190 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 309. Current term is 310. Ops: []
I1014 19:26:37.563132 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 310 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.563315 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 310 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.563429 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 310 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.563794 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 310 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 310 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.563968 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 307.201, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.564188 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 LEADER]: Stepping down as leader of term 309
I1014 19:26:37.564234 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:37.564471 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 309.202, Last appended by leader: 202, Current term: 309, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.564651 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:37.564704 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:37.564828 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 309 FOLLOWER]: Advancing to term 310
I1014 19:26:37.566970 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 310 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.567169 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 310 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.567503 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 310 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 310 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.567591 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 310 election: Election decided. Result: candidate lost.
I1014 19:26:37.567865 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:37.567941 12355 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Leader election lost for term 310. Reason: could not achieve majority
I1014 19:26:37.659523 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:37.659632 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:37.659695 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 309 FOLLOWER]: Advancing to term 310
I1014 19:26:37.661834 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.662344 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 310 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.663015 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 310 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.663090 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 310 candidate_status { last_received { term: 309 index: 202 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.663291 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 310: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:37.663921 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 310 candidate_status { last_received { term: 309 index: 202 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.664094 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (vote granted)
I1014 19:26:37.666234 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 310 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 310.
I1014 19:26:37.667016 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 310 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.667093 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 310 election: Election decided. Result: candidate won.
I1014 19:26:37.667351 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 310 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 310: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:37.667645 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:37.667759 12487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 FOLLOWER]: Leader election won for term 310
I1014 19:26:37.667992 12487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.668254 12487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 309.202, Last appended by leader: 202, Current term: 310, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.669358 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 310 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.669602 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.704071 12355 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.704174 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.200s (starting election)
I1014 19:26:37.704242 12355 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 310 FOLLOWER]: Advancing to term 311
I1014 19:26:37.706346 12355 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 311 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.706892 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 311 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.707108 12355 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 311 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.707572 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 311 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.707746 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 310 FOLLOWER]: Advancing to term 311
I1014 19:26:37.709880 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 311 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 311 because replica has last-logged OpId of term: 309 index: 202, which is greater than that of the candidate, which has last-logged OpId of term: 307 index: 201.
I1014 19:26:37.710202 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 311 candidate_status { last_received { term: 307 index: 201 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.710404 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 311 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.711349 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 311 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 311 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.712106 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 311 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 311 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 311 because replica has last-logged OpId of term: 309 index: 202, which is greater than that of the candidate, which has last-logged OpId of term: 307 index: 201.
I1014 19:26:37.712177 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 311 election: Election decided. Result: candidate lost.
I1014 19:26:37.712426 12355 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.218s (election complete)
I1014 19:26:37.712510 12355 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 311 FOLLOWER]: Leader election lost for term 311. Reason: could not achieve majority
I1014 19:26:37.746341 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 311 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 310. Current term is 311. Ops: []
I1014 19:26:37.746990 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.747262 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 LEADER]: Stepping down as leader of term 310
I1014 19:26:37.747309 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.747531 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 310.203, Last appended by leader: 203, Current term: 310, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.747715 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:37.747776 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:37.747895 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 310 FOLLOWER]: Advancing to term 311
I1014 19:26:37.784912 13232 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 311 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.785012 13232 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:37.785078 13232 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 311 FOLLOWER]: Advancing to term 312
I1014 19:26:37.787286 13232 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.787807 13232 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 312 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.788036 13232 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 312 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:37.788606 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 312 candidate_status { last_received { term: 309 index: 202 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.788802 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 311 FOLLOWER]: Advancing to term 312
I1014 19:26:37.788883 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (vote granted)
I1014 19:26:37.789225 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 312 candidate_status { last_received { term: 309 index: 202 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:37.789391 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 311 FOLLOWER]: Advancing to term 312
I1014 19:26:37.790930 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 312 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 312.
I1014 19:26:37.791550 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 312 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 312 because replica has last-logged OpId of term: 310 index: 203, which is greater than that of the candidate, which has last-logged OpId of term: 309 index: 202.
I1014 19:26:37.792060 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 312 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 312 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 312 because replica has last-logged OpId of term: 310 index: 203, which is greater than that of the candidate, which has last-logged OpId of term: 309 index: 202.
I1014 19:26:37.794909 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 312 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.794991 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 312 election: Election decided. Result: candidate won.
I1014 19:26:37.795258 13232 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:37.795347 13232 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 FOLLOWER]: Leader election won for term 312
I1014 19:26:37.795547 13232 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:37.795778 13232 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 309.202, Last appended by leader: 202, Current term: 312, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.796901 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 312 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.797088 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.823293 11575 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 312 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:37.823438 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:37.823521 11575 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 312 FOLLOWER]: Advancing to term 313
I1014 19:26:37.825778 11575 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.826318 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 313 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.826761 11575 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 313 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:37.827046 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 313 candidate_status { last_received { term: 310 index: 203 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:37.827227 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 312 FOLLOWER]: Advancing to term 313
I1014 19:26:37.827311 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.211s (vote granted)
I1014 19:26:37.829385 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 313 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 313.
I1014 19:26:37.830082 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 313 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:37.830189 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 313 election: Election decided. Result: candidate won.
I1014 19:26:37.830294 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 313 candidate_status { last_received { term: 310 index: 203 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:37.830518 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 313 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.830967 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 313 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 313 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:37.836598 11575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:37.836706 11575 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 FOLLOWER]: Leader election won for term 313
I1014 19:26:37.836923 11575 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:37.837162 11575 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 310.203, Last appended by leader: 203, Current term: 313, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.840879 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.841089 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.906026 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 313 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 312. Current term is 313. Ops: []
I1014 19:26:37.906589 11440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.906841 13232 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 LEADER]: Stepping down as leader of term 312
I1014 19:26:37.906911 13232 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:37.907109 13232 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 312.203, Last appended by leader: 203, Current term: 312, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:37.907263 13232 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:37.907326 13232 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:37.907450 13232 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 312 FOLLOWER]: Advancing to term 313
I1014 19:26:37.927965 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 313 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 312 index: 203. Preceding OpId from leader: term: 313 index: 204. (index mismatch)
I1014 19:26:37.928532 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 202, Time since last communication: 0.000s
I1014 19:26:37.928716 11575 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 312.203 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 202
I1014 19:26:37.929409 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 202
I1014 19:26:37.929461 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 312.203
I1014 19:26:37.929546 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 313 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:37.930162 13232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.930400 13232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.931449 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.931674 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:37.931790 13232 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.932041 13232 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.931910 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.932150 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.932667 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 313 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 307 index: 201. Preceding OpId from leader: term: 313 index: 204. (index mismatch)
I1014 19:26:37.933099 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 201, Time since last communication: 0.000s
I1014 19:26:37.935819 13442 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.935822 12355 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.936081 12355 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.936182 13442 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.936249 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.936466 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:37.936571 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:37.936764 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.022465 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 313 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.022773 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.126581 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 313 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:38.126672 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:38.126746 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 313 FOLLOWER]: Advancing to term 314
I1014 19:26:38.127607 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 313 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:38.127693 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:38.127753 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 313 FOLLOWER]: Advancing to term 314
I1014 19:26:38.128872 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 314 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.129339 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 314 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:38.129842 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.130017 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 314 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.130040 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 314 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:38.130244 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 314: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:38.130337 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 314 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:38.130555 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 314 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.130590 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 314 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 314: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:38.130856 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 314 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.131121 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 314 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.131121 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 314 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.131296 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 314 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.131379 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 313. Current term is 314. Ops: []
I1014 19:26:38.131619 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 314 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:38.131695 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 314 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 314 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.131793 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 314 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 314: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:38.131876 13501 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 313.204, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
I1014 19:26:38.132125 13501 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Stepping down as leader of term 313
I1014 19:26:38.132150 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 314 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 314 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.132189 13501 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:38.132222 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 314 election: Election decided. Result: candidate lost.
I1014 19:26:38.132380 13501 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 313.204, Last appended by leader: 204, Current term: 313, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.132656 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 314 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 314 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 314: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:38.132676 13501 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:38.132732 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 314 election: Election decided. Result: candidate lost.
I1014 19:26:38.132757 13501 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:38.132911 13501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 313 FOLLOWER]: Advancing to term 314
I1014 19:26:38.133000 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:38.133066 11440 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Leader election lost for term 314. Reason: could not achieve majority
I1014 19:26:38.133180 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:38.133235 13444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 314 FOLLOWER]: Leader election lost for term 314. Reason: could not achieve majority
I1014 19:26:38.229393 13501 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 314 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:38.229521 13501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:38.229593 13501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 314 FOLLOWER]: Advancing to term 315
I1014 19:26:38.231902 13501 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.232425 13501 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 315 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:38.233165 13501 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 315 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:38.233160 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 315 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:38.233335 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 314 FOLLOWER]: Advancing to term 315
I1014 19:26:38.233448 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (vote granted)
I1014 19:26:38.233894 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 315 candidate_status { last_received { term: 313 index: 204 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:38.234064 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 314 FOLLOWER]: Advancing to term 315
I1014 19:26:38.234145 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (vote granted)
I1014 19:26:38.235494 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 315 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 315.
I1014 19:26:38.235843 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 315 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:38.235921 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 315 election: Election decided. Result: candidate won.
I1014 19:26:38.236167 13501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:38.236250 13501 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 FOLLOWER]: Leader election won for term 315
I1014 19:26:38.236315 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 315.
I1014 19:26:38.236400 13501 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:38.236690 13501 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 204, Committed index: 204, Last appended: 313.204, Last appended by leader: 204, Current term: 315, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.237905 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 315 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:38.238025 11575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.238236 11575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.325642 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 313 index: 204. Preceding OpId from leader: term: 315 index: 205. (index mismatch)
I1014 19:26:38.326292 11575 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
I1014 19:26:38.327724 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.328028 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.329212 13501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.329499 13501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.329845 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 315 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 313 index: 204. Preceding OpId from leader: term: 315 index: 205. (index mismatch)
I1014 19:26:38.330216 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.330404 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.330955 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 205, Last known committed idx: 204, Time since last communication: 0.000s
I1014 19:26:38.332283 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.332550 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.332784 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.332955 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.419150 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.419334 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:38.485152  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.023s	user 0.001s	sys 0.000s
I1014 19:26:38.485208  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:38.486047  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:38.487063  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 309 vs 315: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.487344  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.487608  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 315 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.520680 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 315 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:38.520762 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:38.520839 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 315 FOLLOWER]: Advancing to term 316
I1014 19:26:38.522997 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.523491 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 316 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:38.523691 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 316 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.524278 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 316 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.524278 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 316 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:38.524500 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 316 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.524518 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 316 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.524884 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 316 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 316 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.530495 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 316 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 316 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.530644 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 316 election: Election decided. Result: candidate lost.
I1014 19:26:38.532169 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:38.532575 13444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Leader election lost for term 316. Reason: could not achieve majority
I1014 19:26:38.536373 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 315. Current term is 316. Ops: []
I1014 19:26:38.536981 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 315.205, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I1014 19:26:38.537211 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 LEADER]: Stepping down as leader of term 315
I1014 19:26:38.537281 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:38.537501 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 315.205, Last appended by leader: 205, Current term: 315, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.537694 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:38.537742 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:38.537868 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 315 FOLLOWER]: Advancing to term 316
I1014 19:26:38.597662 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:38.597776 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:38.597856 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 315 FOLLOWER]: Advancing to term 316
I1014 19:26:38.600147 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.600718 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 316 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:38.601130 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 316 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.601493 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 316 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:38.601747 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 316: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:38.601894 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 316 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.602097 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (vote granted)
I1014 19:26:38.602192 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 316 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 316: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:38.604363 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 316 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 316.
I1014 19:26:38.604861 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 316 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.604934 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 316 election: Election decided. Result: candidate won.
I1014 19:26:38.605171 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:38.605257 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 FOLLOWER]: Leader election won for term 316
I1014 19:26:38.612727 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:38.613055 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 315.205, Last appended by leader: 205, Current term: 316, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.614658 13834 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 316 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.614926 13834 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.615139  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:38.617163  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:38.679596 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:38.679724 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.186s (starting election)
I1014 19:26:38.679801 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 316 FOLLOWER]: Advancing to term 317
I1014 19:26:38.682090 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.682610 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 317 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:38.682816 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 317 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.683682 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 317 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:38.683746 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 317 candidate_status { last_received { term: 315 index: 205 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.683884 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 317 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.683902 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 316 FOLLOWER]: Advancing to term 317
I1014 19:26:38.683995 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:38.684305 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 317 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 317 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.685606 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 317 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 317.
I1014 19:26:38.685981 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 317 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.686059 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 317 election: Election decided. Result: candidate won.
I1014 19:26:38.686298 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:38.686362 13444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 FOLLOWER]: Leader election won for term 317
I1014 19:26:38.686527 13444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:38.686738 13444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 315.205, Last appended by leader: 205, Current term: 317, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.687714 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 317 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.687888 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.688473 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:38.699815 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:38.699995 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 316. Current term is 317. Ops: []
I1014 19:26:38.700608 13834 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I1014 19:26:38.700850 13834 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 317.206 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 205
I1014 19:26:38.700948 11440 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 LEADER]: Stepping down as leader of term 316
I1014 19:26:38.701009 11440 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:38.701207 11440 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 316.206, Last appended by leader: 206, Current term: 316, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.701431 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:38.701484 11440 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:38.701601 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 316 FOLLOWER]: Advancing to term 317
I1014 19:26:38.797307 11440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 317 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:38.797430 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:38.797508 11440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 317 FOLLOWER]: Advancing to term 318
I1014 19:26:38.799753 11440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.800243 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 318 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:38.800599 11440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 318 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.800973 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 318 candidate_status { last_received { term: 316 index: 206 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:38.801158 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 318 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.801290 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 318 candidate_status { last_received { term: 316 index: 206 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:38.801470 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 317 FOLLOWER]: Advancing to term 318
I1014 19:26:38.801568 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (vote granted)
I1014 19:26:38.801631 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 318 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 318 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:38.801913 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 317. Current term is 318. Ops: []
I1014 19:26:38.803740 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 318.
I1014 19:26:38.804122 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 318 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:38.804056 13443 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I1014 19:26:38.804199 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 318 election: Election decided. Result: candidate won.
I1014 19:26:38.804239 13443 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 316.206 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 205
I1014 19:26:38.804353 13443 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Stepping down as leader of term 317
I1014 19:26:38.804410 13443 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:38.804517 11440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:38.804594 11440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 FOLLOWER]: Leader election won for term 318
I1014 19:26:38.804706 11440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:38.804648 13443 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 317.206, Last appended by leader: 206, Current term: 317, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.804863 13443 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:38.804937 13443 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:38.804901 11440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 316.206, Last appended by leader: 206, Current term: 318, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:38.805052 13443 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 317 FOLLOWER]: Advancing to term 318
I1014 19:26:38.806093 13834 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.806272 13834 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.897361 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 315 index: 205. Preceding OpId from leader: term: 318 index: 207. (index mismatch)
I1014 19:26:38.898048 13834 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 205, Time since last communication: 0.000s
I1014 19:26:38.899616 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.899859 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.901492 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 318 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 317 index: 206. Preceding OpId from leader: term: 318 index: 207. (index mismatch)
I1014 19:26:38.902190 11440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 205, Time since last communication: 0.000s
I1014 19:26:38.902408 11440 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 317.206 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 205
I1014 19:26:38.903189 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 205
I1014 19:26:38.903244 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 317.206
I1014 19:26:38.903326 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 318 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:38.903638 13501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.903903 13501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.904031 13501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.904242 13501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.910953 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.911190 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.911406 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.911624 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:38.911741 14045 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.911800 11440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:38.912011 11440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:38.911968 14045 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:26:38.935396 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.509s	user 0.001s	sys 0.000s
I1014 19:26:38.935473 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:38.936137 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:38.936903 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 308 vs 318: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937132 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937302 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937474 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937651 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937801 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.937964 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:38.938118 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.005200 13834 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 318 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.005491 13834 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.005672 14045 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 318 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:39.005774 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:39.005854 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 318 FOLLOWER]: Advancing to term 319
I1014 19:26:39.008155 14045 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.008728 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 319 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.008929 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 319 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.011031 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 319 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.011225 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 319 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.012373 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 319 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 319 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.012693 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 318. Current term is 319. Ops: []
I1014 19:26:39.013013 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 319 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.013190 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 319 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.013734 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 319 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 319 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.013808 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 319 election: Election decided. Result: candidate lost.
I1014 19:26:39.013890 14094 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 318.207, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I1014 19:26:39.014111 14094 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 LEADER]: Stepping down as leader of term 318
I1014 19:26:39.014196 14094 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:39.014376 14094 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 318.207, Last appended by leader: 207, Current term: 318, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.014590 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:39.014638 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:39.014755 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 318 FOLLOWER]: Advancing to term 319
I1014 19:26:39.020944 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:39.021086 14045 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Leader election lost for term 319. Reason: could not achieve majority
I1014 19:26:39.099099 13501 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:39.099228 13501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:39.099301 13501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 318 FOLLOWER]: Advancing to term 319
I1014 19:26:39.101621 13501 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.102202 13501 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 319 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.102890 13501 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 319 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.103248 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 319 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.103512 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 319: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:39.103832 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 319 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.104028 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:39.104120 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 319 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 319: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:39.105777 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 319 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 319.
I1014 19:26:39.106128 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 319 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.106226 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 319 election: Election decided. Result: candidate won.
I1014 19:26:39.106654 13501 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:39.106750 13501 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 FOLLOWER]: Leader election won for term 319
I1014 19:26:39.106937 13501 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:39.107137 13501 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 318.207, Last appended by leader: 207, Current term: 319, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.108192 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 319 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.108381 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.108611 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:39.110082 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:39.136898 14045 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:39.136991 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.172s (starting election)
I1014 19:26:39.137074 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 319 FOLLOWER]: Advancing to term 320
I1014 19:26:39.138602 14045 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.139001 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 320 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.139165 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 320 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.139827 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 320 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.139855 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 320 candidate_status { last_received { term: 318 index: 207 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.139993 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 319 FOLLOWER]: Advancing to term 320
I1014 19:26:39.140053 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 320 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.140082 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:39.140521 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 320 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 320 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.141719 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 320 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 320.
I1014 19:26:39.142154 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 320 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.142232 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 320 election: Election decided. Result: candidate won.
I1014 19:26:39.142526 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.219s (election complete)
I1014 19:26:39.142599 14045 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 FOLLOWER]: Leader election won for term 320
I1014 19:26:39.142714 14045 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:39.142930 14045 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 318.207, Last appended by leader: 207, Current term: 320, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.144780 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.144984 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.185206 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 319. Current term is 320. Ops: []
I1014 19:26:39.185827 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I1014 19:26:39.186074 12487 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 320.208 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 207
I1014 19:26:39.186164 13501 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 LEADER]: Stepping down as leader of term 319
I1014 19:26:39.186228 13501 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:39.186441 13501 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 319.208, Last appended by leader: 208, Current term: 319, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.186637 13501 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:39.186700 13501 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:39.186827 13501 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 319 FOLLOWER]: Advancing to term 320
I1014 19:26:39.222834 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 320 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 318 index: 207. Preceding OpId from leader: term: 320 index: 208. (index mismatch)
I1014 19:26:39.223331 13443 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I1014 19:26:39.224714 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.224925 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.225795 13443 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.226052 13443 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.227092 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 320 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 319 index: 208. Preceding OpId from leader: term: 320 index: 208. (term mismatch)
I1014 19:26:39.227169 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 207
I1014 19:26:39.227205 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 319.208
I1014 19:26:39.227274 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 320 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:39.228039 14045 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I1014 19:26:39.229508 13501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.229756 13501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.229872 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.230069 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.230376 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 320 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.230592 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.329679 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 320 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:39.329790 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:39.329865 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 320 FOLLOWER]: Advancing to term 321
I1014 19:26:39.332072 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.332645 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 321 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.332842 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 321 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.333448 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 321 candidate_status { last_received { term: 320 index: 208 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.333669 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 321 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.333962 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 321 candidate_status { last_received { term: 320 index: 208 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.333962 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 320 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:39.334153 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:39.334229 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 320 FOLLOWER]: Advancing to term 321
I1014 19:26:39.334635 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 320. Current term is 321. Ops: []
I1014 19:26:39.335165 14045 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 320.208, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I1014 19:26:39.335384 14045 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Stepping down as leader of term 320
I1014 19:26:39.335436 14045 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:39.335659 14045 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 320.208, Last appended by leader: 208, Current term: 320, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.335850 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:39.335911 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:39.336066 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 FOLLOWER]: Advancing to term 321
I1014 19:26:39.336519 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.336947 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 321 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.337142 12487 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 321 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.337435 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 321: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:39.337653 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 321 candidate_status { last_received { term: 320 index: 208 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.337887 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 321: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:39.337967 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 321 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 321: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:39.338302 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 321 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 321: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:39.338639 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 321 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 320 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 321 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.338615 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 321 candidate_status { last_received { term: 320 index: 208 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.338726 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 321 election: Election decided. Result: candidate lost.
I1014 19:26:39.338748 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (vote granted)
I1014 19:26:39.340564 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 321.
I1014 19:26:39.340765 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:39.340852 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 321 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.340853 14094 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Leader election lost for term 321. Reason: could not achieve majority
I1014 19:26:39.340946 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 321 election: Election decided. Result: candidate won.
I1014 19:26:39.341182 12487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:39.341295 12487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 FOLLOWER]: Leader election won for term 321
I1014 19:26:39.341524 12487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:39.341753 12487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 320.208, Last appended by leader: 208, Current term: 321, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.345959 13501 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.346215 13501 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.430631 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 320 index: 208. Preceding OpId from leader: term: 321 index: 209. (index mismatch)
I1014 19:26:39.431147 12487 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I1014 19:26:39.432478 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.432693 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.433892 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.434134 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.435364 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.435657 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.436411 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 320 index: 208. Preceding OpId from leader: term: 321 index: 209. (index mismatch)
I1014 19:26:39.436841 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I1014 19:26:39.438235 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.438431 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.438592 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.438803 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.514209 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.514895 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.550974  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:39.551964  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:39.553546  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 316 vs 321: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:39.553848  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:39.554101  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:39.554337  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 321 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.631227 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:39.631338 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (starting election)
I1014 19:26:39.631412 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 321 FOLLOWER]: Advancing to term 322
I1014 19:26:39.633038 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.633535 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 322 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.633754 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 322 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.634392 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 322 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.634387 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 322 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.634613 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 322 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.634624 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 322 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.635008 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 322 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 322 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.635144 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 322 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 322 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.635227 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 322 election: Election decided. Result: candidate lost.
I1014 19:26:39.635488 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:39.635609 14094 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Leader election lost for term 322. Reason: could not achieve majority
I1014 19:26:39.653635 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 321. Current term is 322. Ops: []
I1014 19:26:39.654902 12487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 321.209, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I1014 19:26:39.655176 12487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 LEADER]: Stepping down as leader of term 321
I1014 19:26:39.655241 12487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:39.655470 12487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 321.209, Last appended by leader: 209, Current term: 321, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.655673 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:39.655732 12487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:39.655896 12487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 321 FOLLOWER]: Advancing to term 322
I1014 19:26:39.698098 14045 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:39.698240 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:39.698323 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 321 FOLLOWER]: Advancing to term 322
I1014 19:26:39.699872 14045 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.700359 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 322 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.700573 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 322 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.701074 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 322 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.701210 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 322 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.701290 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 322: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:39.701395 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:39.701719 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 322 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 322: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:39.703456 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 322 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 322.
I1014 19:26:39.703788 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 322 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.703869 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 322 election: Election decided. Result: candidate won.
I1014 19:26:39.704138 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:39.704213 14045 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 FOLLOWER]: Leader election won for term 322
I1014 19:26:39.704330 14045 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:39.704587 14045 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 321.209, Last appended by leader: 209, Current term: 322, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.705590 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 322 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.705830 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.777829 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:39.777921 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.202s (starting election)
I1014 19:26:39.777983 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 322 FOLLOWER]: Advancing to term 323
I1014 19:26:39.779999 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.780535 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 323 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.780719 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 323 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.781658 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 323 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.781873 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 323 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.781872 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 323 candidate_status { last_received { term: 321 index: 209 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.782068 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 322 FOLLOWER]: Advancing to term 323
I1014 19:26:39.782169 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:39.782236 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 323 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 323 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.784433 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 323 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 323.
I1014 19:26:39.784827 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 323 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.784915 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 323 election: Election decided. Result: candidate won.
I1014 19:26:39.785346 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:39.785432 14094 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 FOLLOWER]: Leader election won for term 323
I1014 19:26:39.794396 14094 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:39.794734 14094 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 321.209, Last appended by leader: 209, Current term: 323, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.796382 14356 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.796633 14356 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.800339  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:39.802529  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:39.807046 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 322. Current term is 323. Ops: []
I1014 19:26:39.807643 13444 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I1014 19:26:39.807907 13444 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 323.210 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 209
I1014 19:26:39.808037 14045 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 LEADER]: Stepping down as leader of term 322
I1014 19:26:39.808099 14045 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:39.808305 14045 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 322.210, Last appended by leader: 210, Current term: 322, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.808532 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:39.808598 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:39.808723 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 322 FOLLOWER]: Advancing to term 323
I1014 19:26:39.871807 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 323 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 321 index: 209. Preceding OpId from leader: term: 323 index: 210. (index mismatch)
I1014 19:26:39.872378 14356 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I1014 19:26:39.873704 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.873997 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.874897 14356 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.875133 14356 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:39.875890 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 323 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 322 index: 210. Preceding OpId from leader: term: 323 index: 210. (term mismatch)
I1014 19:26:39.875954 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 209
I1014 19:26:39.876001 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 322.210
I1014 19:26:39.876058 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 323 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:39.876204 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.876389 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.877355 14356 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I1014 19:26:39.878674 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.878916 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.879086 14045 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 323 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:39.879302 14045 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:39.975999 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 323 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:39.976097 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:39.976176 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 323 FOLLOWER]: Advancing to term 324
I1014 19:26:39.978363 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.978905 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 324 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:39.979122 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 324 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.979298 14045 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 323 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:39.979421 14045 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:39.979533 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 323 FOLLOWER]: Advancing to term 324
I1014 19:26:39.979853 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 324 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:39.980183 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 324 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.980381 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 324 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.980767 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 324 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 324 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.981891 14045 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 324 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.982240 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 324 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 324: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:39.982414 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 324 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:39.982720 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 324 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 324 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 324: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:39.982745 14045 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 324 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:39.982810 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 324 election: Election decided. Result: candidate lost.
I1014 19:26:39.983073 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:39.983053 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 324 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:39.983191 14227 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Leader election lost for term 324. Reason: could not achieve majority
I1014 19:26:39.983217 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 324 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.983371 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 324 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:39.983592 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 324: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:39.983989 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 324 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 324: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:39.984120 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 324 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 324 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:39.984200 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 324 election: Election decided. Result: candidate lost.
I1014 19:26:39.984251 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 323. Current term is 324. Ops: []
I1014 19:26:39.984498 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:26:39.984599 13444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 324 FOLLOWER]: Leader election lost for term 324. Reason: could not achieve majority
I1014 19:26:39.984925 14358 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 323.210, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I1014 19:26:39.985160 14094 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Stepping down as leader of term 323
I1014 19:26:39.985206 14094 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:39.985383 14094 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 323.210, Last appended by leader: 210, Current term: 323, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:39.985554 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:39.985592 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:39.985688 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 323 FOLLOWER]: Advancing to term 324
I1014 19:26:40.074287 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 324 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:40.074400 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:40.074498 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 324 FOLLOWER]: Advancing to term 325
I1014 19:26:40.076102 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.076531 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 325 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.076725 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 325 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.077231 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 325 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.077437 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 324 FOLLOWER]: Advancing to term 325
I1014 19:26:40.077560 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.151s (vote granted)
I1014 19:26:40.077693 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 325 candidate_status { last_received { term: 323 index: 210 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.077891 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 324 FOLLOWER]: Advancing to term 325
I1014 19:26:40.077986 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.173s (vote granted)
I1014 19:26:40.079784 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 325.
I1014 19:26:40.080165 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 325 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 325.
I1014 19:26:40.080557 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 325 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.080634 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 325 election: Election decided. Result: candidate won.
I1014 19:26:40.081331 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 325 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.081792 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:40.081881 14094 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 FOLLOWER]: Leader election won for term 325
I1014 19:26:40.081995 14094 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:40.082214 14094 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 323.210, Last appended by leader: 210, Current term: 325, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.082996 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.083210 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.089640 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:40.090317 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:40.091117 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 319 vs 325: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 325 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.091348 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 325 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.091576 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 325 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.091753 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 325 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.187594 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 323 index: 210. Preceding OpId from leader: term: 325 index: 211. (index mismatch)
I1014 19:26:40.188190 14358 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I1014 19:26:40.189829 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.190083 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.190738 14356 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.191017 14356 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.191486 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.191682 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.192270 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 325 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 323 index: 210. Preceding OpId from leader: term: 325 index: 211. (index mismatch)
I1014 19:26:40.194538 14356 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I1014 19:26:40.199707 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.199955 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.207530 14488 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.207770 14488 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.295915 14488 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 325 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:40.296022 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:26:40.296111 14488 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 325 FOLLOWER]: Advancing to term 326
I1014 19:26:40.298370 14488 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.298924 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 326 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.299093 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 326 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.299690 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 326 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.299698 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 326 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.299919 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 326 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.299919 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 326 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.300351 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 326 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 326 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.300549 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 326 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 326 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.300624 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 326 election: Election decided. Result: candidate lost.
I1014 19:26:40.300916 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:40.300993 14488 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Leader election lost for term 326. Reason: could not achieve majority
I1014 19:26:40.302901 14358 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.303107 14358 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.304095 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 325. Current term is 326. Ops: []
I1014 19:26:40.304891 14094 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 325.211, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I1014 19:26:40.305097 14094 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 LEADER]: Stepping down as leader of term 325
I1014 19:26:40.305142 14094 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:40.305316 14094 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 325.211, Last appended by leader: 211, Current term: 325, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.305518 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:40.305572 14094 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:40.305685 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 325 FOLLOWER]: Advancing to term 326
W1014 19:26:40.317144 13891 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.617s	user 0.001s	sys 0.000s
I1014 19:26:40.317219 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:40.318140 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:40.319257 13891 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 317 vs 325: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.319586 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.319838 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.320353 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 325 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.382820 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:40.382917 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:40.382999 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 325 FOLLOWER]: Advancing to term 326
I1014 19:26:40.384619 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.385043 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 326 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.385251 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 326 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.385737 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 326 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.385942 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (vote granted)
I1014 19:26:40.385937 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 326 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.386152 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 326: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:40.386618 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 326 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 326: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:40.387583 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 326 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 326.
I1014 19:26:40.387866 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 326 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.387944 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 326 election: Election decided. Result: candidate won.
I1014 19:26:40.388164 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:40.388238 13444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 FOLLOWER]: Leader election won for term 326
I1014 19:26:40.388381 13444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:40.388569 13444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 325.211, Last appended by leader: 211, Current term: 326, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.389585 14045 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 326 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.389755 14045 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.389955 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:40.391355 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:40.406333 14488 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:40.406424 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (starting election)
I1014 19:26:40.406527 14488 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 326 FOLLOWER]: Advancing to term 327
I1014 19:26:40.408157 14488 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.408679 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 327 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.408840 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 327 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.409523 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 327 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.409689 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 326 FOLLOWER]: Advancing to term 327
I1014 19:26:40.409773 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (vote granted)
I1014 19:26:40.411830 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 327.
I1014 19:26:40.412093 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 327 candidate_status { last_received { term: 325 index: 211 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.412282 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 327 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.412719 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 327 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 327 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.413422 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 327 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.413498 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 327 election: Election decided. Result: candidate won.
I1014 19:26:40.413763 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:40.413831 14488 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 FOLLOWER]: Leader election won for term 327
I1014 19:26:40.413978 14488 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:40.414187 14488 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 325.211, Last appended by leader: 211, Current term: 327, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.416533 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.416760 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.416975 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:40.418879 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:40.466042 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 326. Current term is 327. Ops: []
I1014 19:26:40.466703 14045 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I1014 19:26:40.466953 14045 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 LEADER]: Stepping down as leader of term 326
I1014 19:26:40.467012 14045 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:40.467213 14045 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 326.212, Last appended by leader: 212, Current term: 326, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.467409 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:40.467458 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:40.467614 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 326 FOLLOWER]: Advancing to term 327
I1014 19:26:40.494921 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 327 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 326 index: 212. Preceding OpId from leader: term: 327 index: 212. (term mismatch)
I1014 19:26:40.494997 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 211
I1014 19:26:40.495031 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 326.212
I1014 19:26:40.495110 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 327 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:40.495786 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I1014 19:26:40.496949 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 325 index: 211. Preceding OpId from leader: term: 327 index: 212. (index mismatch)
I1014 19:26:40.497146 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.497391 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.497424 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 212, Last known committed idx: 211, Time since last communication: 0.000s
I1014 19:26:40.498595 14358 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.498646 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.498823 14358 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.498881 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.499604 14488 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.499783 14488 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.500098 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.500315 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.500866 14358 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.501046 14358 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.600605 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 327 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:40.600703 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:40.600778 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 327 FOLLOWER]: Advancing to term 328
I1014 19:26:40.602917 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.603415 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 328 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.603744 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 328 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.604362 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 328 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.604353 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 328 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.604574 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 328 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.604578 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 328 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.604940 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 328 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 328 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.605113 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 328 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 328 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.605247 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 328 election: Election decided. Result: candidate lost.
I1014 19:26:40.605533 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (election complete)
I1014 19:26:40.605600 13444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Leader election lost for term 328. Reason: could not achieve majority
I1014 19:26:40.610458  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:40.611400  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:40.612550  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 323 vs 327: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.612987  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.613312  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.613613  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:40.613898  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 327 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.622915 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 327. Current term is 328. Ops: []
I1014 19:26:40.623512 14227 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 327.212, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I1014 19:26:40.623800 14488 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 LEADER]: Stepping down as leader of term 327
I1014 19:26:40.623864 14488 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:40.624135 14488 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 327.212, Last appended by leader: 212, Current term: 327, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.624374 14488 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:40.624444 14488 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:40.624629 14488 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 327 FOLLOWER]: Advancing to term 328
I1014 19:26:40.692585 14358 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:40.692687 14358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:40.692757 14358 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 327 FOLLOWER]: Advancing to term 328
I1014 19:26:40.695080 14358 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.695675 14358 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 328 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.695907 14358 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 328 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.696557 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 328 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.696651 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 328 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.696748 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 328: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:40.696831 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:26:40.697132 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 328 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 328: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:40.699101 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 328 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 328.
I1014 19:26:40.699503 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 328 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.699599 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 328 election: Election decided. Result: candidate won.
I1014 19:26:40.699883 14358 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:40.699956 14358 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 FOLLOWER]: Leader election won for term 328
I1014 19:26:40.700165 14358 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:40.700430 14358 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 327.212, Last appended by leader: 212, Current term: 328, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.701829 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 328 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.702018 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.702239  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:40.703881  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:40.707933 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:40.708037 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:40.708133 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 328 FOLLOWER]: Advancing to term 329
I1014 19:26:40.710340 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.710819 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 329 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.711043 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 329 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.711613 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 329 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.711817 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 329 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.711715 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 329 candidate_status { last_received { term: 327 index: 212 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.711916 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 328 FOLLOWER]: Advancing to term 329
I1014 19:26:40.712003 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (vote granted)
I1014 19:26:40.712262 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 329 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 329 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.713587 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 329 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 329.
I1014 19:26:40.713891 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 329 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.713954 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 329 election: Election decided. Result: candidate won.
I1014 19:26:40.714200 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.161s (election complete)
I1014 19:26:40.714277 13444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 FOLLOWER]: Leader election won for term 329
I1014 19:26:40.714422 13444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:40.714656 13444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 327.212, Last appended by leader: 212, Current term: 329, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.716265 14045 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.716569 14045 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.823341 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 328. Current term is 329. Ops: []
I1014 19:26:40.823350 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 329 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 328. Current term is 329. Ops: []
I1014 19:26:40.823849 28826 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 LEADER]: Stepping down as leader of term 328
I1014 19:26:40.823982 28826 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:40.823974 14094 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I1014 19:26:40.824187 14094 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 329.213 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 212
I1014 19:26:40.824506 28826 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 328.213, Last appended by leader: 213, Current term: 328, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.824827 28826 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:40.824831 14358 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I1014 19:26:40.825099 28826 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:40.825245 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 328 FOLLOWER]: Advancing to term 329
I1014 19:26:40.827610 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 329 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 328 index: 213. Preceding OpId from leader: term: 329 index: 213. (term mismatch)
I1014 19:26:40.827667 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 212
I1014 19:26:40.827731 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 328.213
I1014 19:26:40.827805 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 329 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:40.828326 14045 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I1014 19:26:40.829478 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.829701 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.830596 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.830813 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:40.831023 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 329 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 327 index: 212. Preceding OpId from leader: term: 329 index: 213. (index mismatch)
I1014 19:26:40.831609 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.831820 13444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I1014 19:26:40.831884 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.833106 14488 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.833463 14488 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.833626 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 329 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:40.833928 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:40.931686 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 329 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:40.931787 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (starting election)
I1014 19:26:40.931870 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 329 FOLLOWER]: Advancing to term 330
I1014 19:26:40.933475 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 329 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:40.933562 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:40.933639 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 329 FOLLOWER]: Advancing to term 330
I1014 19:26:40.934085 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.934618 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 330 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.934823 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 330 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:40.935282 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 330 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.935519 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 330 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.935477 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 330 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:40.935830 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 330 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.936054 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 330 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 330 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.936379 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 330 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 330: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:40.936370 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 330 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:40.936522 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 329. Current term is 330. Ops: []
I1014 19:26:40.936861 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 330 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:40.937014 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 330 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 330 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 330: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:40.937100 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 330 election: Election decided. Result: candidate lost.
I1014 19:26:40.937153 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 330 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:40.937151 13444 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 329.213, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I1014 19:26:40.937485 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 330 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.937580 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:40.937681 14094 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Leader election lost for term 330. Reason: could not achieve majority
I1014 19:26:40.937728 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 330 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:40.937795 13444 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Stepping down as leader of term 329
I1014 19:26:40.937846 13444 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:40.937918 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 330: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:40.938045 13444 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 329.213, Last appended by leader: 213, Current term: 329, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:40.938247 13444 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:40.938338 13444 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:40.938326 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 330 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 330: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:40.938463 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 FOLLOWER]: Advancing to term 330
I1014 19:26:40.938524 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 330 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 329 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 330 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:40.938596 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 330 election: Election decided. Result: candidate lost.
I1014 19:26:40.938829 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:40.938894 14227 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 330 FOLLOWER]: Leader election lost for term 330. Reason: could not achieve majority
I1014 19:26:41.018074 13444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 330 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:41.018196 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:41.018313 13444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 330 FOLLOWER]: Advancing to term 331
I1014 19:26:41.024989 13444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.027196 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 331 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.027487 13444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 331 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.028158 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 331 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.028339 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 330 FOLLOWER]: Advancing to term 331
I1014 19:26:41.028448 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.175s (vote granted)
I1014 19:26:41.030705 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 331.
I1014 19:26:41.031044 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 331 candidate_status { last_received { term: 329 index: 213 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.031174 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 331 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.031225 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 330 FOLLOWER]: Advancing to term 331
I1014 19:26:41.031252 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 331 election: Election decided. Result: candidate won.
I1014 19:26:41.031314 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.202s (vote granted)
I1014 19:26:41.031530 13444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:41.031603 13444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 FOLLOWER]: Leader election won for term 331
I1014 19:26:41.031723 13444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:41.031975 13444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 329.213, Last appended by leader: 213, Current term: 331, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.033023 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.033236 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.033694 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 331 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 331.
I1014 19:26:41.034400 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 331 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.131455 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 329 index: 213. Preceding OpId from leader: term: 331 index: 214. (index mismatch)
I1014 19:26:41.131976 14045 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I1014 19:26:41.133044 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.133225 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.134318 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.134537 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.134840 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 331 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 329 index: 213. Preceding OpId from leader: term: 331 index: 214. (index mismatch)
I1014 19:26:41.134892 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.135069 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.135527 13444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s
I1014 19:26:41.136939 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.137151 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.137282 14488 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.137424 14488 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.236742 14488 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 331 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:41.236874 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:41.237120 14488 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 331 FOLLOWER]: Advancing to term 332
I1014 19:26:41.239600 14488 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.240025 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 331. Current term is 332. Ops: []
I1014 19:26:41.240231 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 332 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.241359 14488 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 332 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.241508 13444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 331 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.241852 13444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.241861 14045 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 331.214, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I1014 19:26:41.242128 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 332 candidate_status { last_received { term: 331 index: 214 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.242177 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 332 candidate_status { last_received { term: 331 index: 214 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.242314 14045 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 LEADER]: Stepping down as leader of term 331
I1014 19:26:41.242352 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 332 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.242414 14045 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:41.242660 14045 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 331.214, Last appended by leader: 214, Current term: 331, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.242859 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:41.242933 14045 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:41.242956 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 332 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 332 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.243088 14045 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 331 FOLLOWER]: Advancing to term 332
I1014 19:26:41.245290 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (vote granted)
I1014 19:26:41.247309 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 332.
I1014 19:26:41.247802 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 332 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.247885 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 332 election: Election decided. Result: candidate won.
I1014 19:26:41.248144 14488 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:41.248219 14488 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 FOLLOWER]: Leader election won for term 332
I1014 19:26:41.248374 14488 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:41.248617 14488 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 331.214, Last appended by leader: 214, Current term: 332, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.249788 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.249948 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.329553 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:41.329650 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (starting election)
I1014 19:26:41.329735 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 331 FOLLOWER]: Advancing to term 332
I1014 19:26:41.331962 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 332 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.332542 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 332 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.333770 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 332 candidate_status { last_received { term: 331 index: 214 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.333973 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 332: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:41.334383 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 332 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 332: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:41.334740 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 332 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.335095 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 332 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 331 index: 214. Preceding OpId from leader: term: 332 index: 215. (index mismatch)
I1014 19:26:41.335302 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 332 candidate_status { last_received { term: 331 index: 214 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.335561 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 332 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.335732 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I1014 19:26:41.336040 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 332 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 332 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.336131 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 332 election: Election decided. Result: candidate lost.
I1014 19:26:41.336470 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:41.336563 14094 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 332 FOLLOWER]: Leader election lost for term 332. Reason: could not achieve majority
I1014 19:26:41.337167 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.338177 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.338397 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.338711 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.339146 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 331 index: 214. Preceding OpId from leader: term: 332 index: 215. (index mismatch)
I1014 19:26:41.339673 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I1014 19:26:41.340231 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.340744 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.341313 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.341476 14045 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.341766 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.341768 14045 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.440696 14094 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 332 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:41.440785 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:41.440851 14094 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 332 FOLLOWER]: Advancing to term 333
I1014 19:26:41.441299 14791 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:41.441375 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:41.441445 14791 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 332 FOLLOWER]: Advancing to term 333
I1014 19:26:41.443601 14791 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.444087 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 333 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.444244 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 333 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.444990 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 333 candidate_status { last_received { term: 332 index: 215 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.445230 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 333 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.445307 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 333 candidate_status { last_received { term: 332 index: 215 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.445653 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 333 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 333 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.449342 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 332 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.449726 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.449829 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 332. Current term is 333. Ops: []
I1014 19:26:41.450368 14877 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 332.215, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I1014 19:26:41.450667 14227 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Stepping down as leader of term 332
I1014 19:26:41.450721 14227 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:41.450959 14227 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 332.215, Last appended by leader: 215, Current term: 332, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.451128 14227 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:41.451194 14227 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:41.451318 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 332 FOLLOWER]: Advancing to term 333
I1014 19:26:41.453754 14094 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.454118 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 333: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:41.454759 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 333 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 333: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:41.454849 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 333 election: Election decided. Result: candidate lost.
I1014 19:26:41.455543 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:41.455646 14791 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Leader election lost for term 333. Reason: could not achieve majority
I1014 19:26:41.456221 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 333 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.456411 14094 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 333 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.456859 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 333 candidate_status { last_received { term: 332 index: 215 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.457041 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 333: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.457561 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 333 candidate_status { last_received { term: 332 index: 215 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.457669 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 333 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 333: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.457734 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:41.459308 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 333 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 333.
I1014 19:26:41.459705 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 333 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.459762 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 333 election: Election decided. Result: candidate won.
I1014 19:26:41.460093 14094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:41.460362 14094 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 FOLLOWER]: Leader election won for term 333
I1014 19:26:41.460702 14094 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:41.461513 14094 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 332.215, Last appended by leader: 215, Current term: 333, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.463006 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.463176 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.483682  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:41.484537  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:41.486574 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 332 index: 215. Preceding OpId from leader: term: 333 index: 217. (index mismatch)
I1014 19:26:41.486600 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 333 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 332 index: 215. Preceding OpId from leader: term: 333 index: 217. (index mismatch)
I1014 19:26:41.487064 14878 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I1014 19:26:41.488562 14878 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I1014 19:26:41.488996 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.489220 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.494967 14877 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.495323 14877 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.495563 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.495740 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.497197 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.497381 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.497611 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.497745 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.497905  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:41.499240  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:41.499840 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 333 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.500257 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.590782 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 333 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:41.590888 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (starting election)
I1014 19:26:41.590971 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 333 FOLLOWER]: Advancing to term 334
I1014 19:26:41.591576 14791 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:41.591650 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:41.591706 14791 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 333 FOLLOWER]: Advancing to term 334
I1014 19:26:41.592808 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 334 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.593252 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 334 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.593475 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 334 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.593423 14791 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.593909 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 334 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.593976 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 334 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.594116 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 334 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.594233 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 334: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.594609 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 334 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.594697 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 334 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.594843 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 334 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.595036 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 334 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.595140 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 334 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:41.595273 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 334 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 334 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.595737 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 334 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 334: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:41.595906 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 334 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 334: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.596122 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 334 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 334 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:41.596191 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 334 election: Election decided. Result: candidate lost.
I1014 19:26:41.596573 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 334 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 334 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 334: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:41.596647 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 334 election: Election decided. Result: candidate lost.
I1014 19:26:41.596890 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:41.596954 14791 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Leader election lost for term 334. Reason: could not achieve majority
I1014 19:26:41.597245 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:41.597313 14227 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 334 FOLLOWER]: Leader election lost for term 334. Reason: could not achieve majority
I1014 19:26:41.600522 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 333. Current term is 334. Ops: []
I1014 19:26:41.601008 14879 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 333.217, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I1014 19:26:41.601217 14878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Stepping down as leader of term 333
I1014 19:26:41.601258 14878 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:41.601431 14878 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 333.217, Last appended by leader: 217, Current term: 333, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.601600 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:41.601639 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:41.601768 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 333 FOLLOWER]: Advancing to term 334
W1014 19:26:41.648346 13891 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.257s	user 0.010s	sys 0.001s
I1014 19:26:41.648394 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:41.649168 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:41.649924 13891 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 326 vs 334: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:41.650121 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:41.650260 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:41.650398 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:41.650559 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:41.650694 13891 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 334 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.680532 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 334 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:41.680608 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:41.680655 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 334 FOLLOWER]: Advancing to term 335
I1014 19:26:41.682708 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.683193 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 335 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.683342 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 335 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.683842 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 335 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.683990 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 334 FOLLOWER]: Advancing to term 335
I1014 19:26:41.684048 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.157s (vote granted)
I1014 19:26:41.686131 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 335 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 335.
I1014 19:26:41.686483 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 335 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.686445 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 335 candidate_status { last_received { term: 333 index: 217 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.686565 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 335 election: Election decided. Result: candidate won.
I1014 19:26:41.686625 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 334 FOLLOWER]: Advancing to term 335
I1014 19:26:41.686707 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:41.686751 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:41.686807 14878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 FOLLOWER]: Leader election won for term 335
I1014 19:26:41.686980 14878 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:41.687145 14878 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 333.217, Last appended by leader: 217, Current term: 335, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.688438 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.688714 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.688875 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 335.
I1014 19:26:41.689268 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 335 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.705929  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:41.706748  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:41.708631 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 335 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 333 index: 217. Preceding OpId from leader: term: 335 index: 219. (index mismatch)
I1014 19:26:41.708734 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 333 index: 217. Preceding OpId from leader: term: 335 index: 219. (index mismatch)
I1014 19:26:41.709095 14878 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I1014 19:26:41.709249 14879 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I1014 19:26:41.710759 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.710966 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.710942 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.711123 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.712383 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.712419 14094 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.712615 14094 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.712576 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.713392 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.713636 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.714079 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 335 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.714262 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:41.714632  7718 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:41.715984  7718 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:41.911687 14791 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 335 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:41.911775 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:41.911830 14791 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 335 FOLLOWER]: Advancing to term 336
I1014 19:26:41.914094 14791 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.914396 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 335. Current term is 336. Ops: []
I1014 19:26:41.914625 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 336 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.914938 14791 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 336 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:41.914968 14878 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 335.219, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I1014 19:26:41.915227 14878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 LEADER]: Stepping down as leader of term 335
I1014 19:26:41.915271 14878 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:41.915233 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 336 candidate_status { last_received { term: 335 index: 219 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.915460 14878 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 335.219, Last appended by leader: 219, Current term: 335, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.915585 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:41.915608 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:41.915688 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 335 FOLLOWER]: Advancing to term 336
I1014 19:26:41.915866 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 336 candidate_status { last_received { term: 335 index: 219 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:41.916083 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 336 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.916715 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 336 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 336 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.917803 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (vote granted)
I1014 19:26:41.919664 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 336.
I1014 19:26:41.920014 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 336 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.920074 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 336 election: Election decided. Result: candidate won.
I1014 19:26:41.920347 14791 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:41.920410 14791 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 FOLLOWER]: Leader election won for term 336
I1014 19:26:41.920732 14791 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:41.920927 14791 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 335.219, Last appended by leader: 219, Current term: 336, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.921900 15016 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:41.922031 15016 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:41.922457 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:41.923806 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:41.984064 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:41.984151 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:26:41.984242 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 335 FOLLOWER]: Advancing to term 336
I1014 19:26:41.985983 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 336 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:41.986443 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 336 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:41.986608 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 336 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:41.987223 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 336 candidate_status { last_received { term: 335 index: 219 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:41.987468 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 336: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.987385 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 336 candidate_status { last_received { term: 335 index: 219 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:41.987582 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 336 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.987870 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 336 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 336: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:41.988050 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 336 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 336 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:41.988128 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 336 election: Election decided. Result: candidate lost.
I1014 19:26:41.988355 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:41.988488 14227 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 336 FOLLOWER]: Leader election lost for term 336. Reason: could not achieve majority
I1014 19:26:42.004283 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 336 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 335 index: 219. Preceding OpId from leader: term: 336 index: 220. (index mismatch)
I1014 19:26:42.004772 15016 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I1014 19:26:42.006108 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.006341 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.007057 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.007241 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.007684 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.007863 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.008222 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 335 index: 219. Preceding OpId from leader: term: 336 index: 220. (index mismatch)
I1014 19:26:42.008647 15016 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I1014 19:26:42.010031 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.010013 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.010196 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.010197 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.097894 15016 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 336 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.098104 15016 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.184028 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 336 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:42.184118 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:42.184171 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 336 FOLLOWER]: Advancing to term 337
I1014 19:26:42.186455 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.186954 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 337 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.187108 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 337 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:42.187804 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 337 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:42.187825 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 337 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:42.187988 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 337 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.188000 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 337 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.188421 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 337 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 337 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.188654 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 337 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 337 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.188715 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 337 election: Election decided. Result: candidate lost.
I1014 19:26:42.188894 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:42.188951 14227 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Leader election lost for term 337. Reason: could not achieve majority
I1014 19:26:42.197808 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 336. Current term is 337. Ops: []
I1014 19:26:42.198719 14791 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 336.220, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I1014 19:26:42.198971 15142 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Stepping down as leader of term 336
I1014 19:26:42.199010 15142 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:42.199187 15142 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 336.220, Last appended by leader: 220, Current term: 336, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.199403 15142 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:42.199465 15142 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:42.199587 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 336 FOLLOWER]: Advancing to term 337
I1014 19:26:42.288023 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:42.288122 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:42.288182 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 336 FOLLOWER]: Advancing to term 337
I1014 19:26:42.290112 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.290616 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 337 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.290767 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 337 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.291671 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 337 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:42.291882 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 337: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:42.292423 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 337 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:42.292598 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:42.294526 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 337 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 337.
I1014 19:26:42.295233 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 337 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.295292 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 337 election: Election decided. Result: candidate won.
I1014 19:26:42.295729 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 337 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 337: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:42.296030 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
W1014 19:26:42.296072 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.877s	user 0.002s	sys 0.000s
I1014 19:26:42.296105 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:42.296093 14878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 FOLLOWER]: Leader election won for term 337
I1014 19:26:42.296392 14878 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:42.296667 14878 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 336.220, Last appended by leader: 220, Current term: 337, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.297029 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:42.298128 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 337 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.298326 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.298822 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 327 vs 337: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:42.299093 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:42.299329 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:42.299582 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:42.299808 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:42.300024 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 337 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.336205 14227 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:42.336292 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.203s (starting election)
I1014 19:26:42.336349 14227 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 337 FOLLOWER]: Advancing to term 338
I1014 19:26:42.338304 14227 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.338811 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 338 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.338963 14227 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 338 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:42.339573 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 338 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:42.339591 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 338 candidate_status { last_received { term: 336 index: 220 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:42.339716 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 337 FOLLOWER]: Advancing to term 338
I1014 19:26:42.339763 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 338 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.339771 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:42.340157 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 338 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 338 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.341861 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 338 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 338.
I1014 19:26:42.342332 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 338 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.342382 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 338 election: Election decided. Result: candidate won.
I1014 19:26:42.342743 14227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:42.342799 14227 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 FOLLOWER]: Leader election won for term 338
I1014 19:26:42.343045 14227 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:42.343286 14227 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 336.220, Last appended by leader: 220, Current term: 338, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.344657 15158 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.344846 15158 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.345053 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:42.346887 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:42.388648 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 338 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 337. Current term is 338. Ops: []
I1014 19:26:42.389241 14878 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I1014 19:26:42.389526 14879 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 LEADER]: Stepping down as leader of term 337
I1014 19:26:42.389566 14879 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:42.389727 14879 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 337.221, Last appended by leader: 221, Current term: 337, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.389860 14879 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:42.389905 14879 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:42.389994 14879 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 337 FOLLOWER]: Advancing to term 338
I1014 19:26:42.433914 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 337 index: 221. Preceding OpId from leader: term: 338 index: 221. (term mismatch)
I1014 19:26:42.433964 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 220
I1014 19:26:42.433984 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 337.221
I1014 19:26:42.434017 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:42.435618 15158 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I1014 19:26:42.436745 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.436933 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.438906 14227 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.439088 14227 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.439864 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 338 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 336 index: 220. Preceding OpId from leader: term: 338 index: 221. (index mismatch)
I1014 19:26:42.439798 14879 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.439957 14879 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.440498 14227 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I1014 19:26:42.441828 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.441861 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.442029 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.441995 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.528621 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 338 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.528842 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.629798 15142 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 338 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:42.629882 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:42.629956 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 338 FOLLOWER]: Advancing to term 339
I1014 19:26:42.632103 15142 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.632622 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 339 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:42.632779 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 339 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.633507 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 339 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:42.633703 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 339 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.634007 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 339 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:42.634191 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 339 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.634531 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 339 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 339 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.634785 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 339 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 339 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.634841 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 339 election: Election decided. Result: candidate lost.
I1014 19:26:42.635164 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:42.635311 15142 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Leader election lost for term 339. Reason: could not achieve majority
I1014 19:26:42.642470 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 338. Current term is 339. Ops: []
I1014 19:26:42.643049 15158 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 338.221, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I1014 19:26:42.643293 15158 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 LEADER]: Stepping down as leader of term 338
I1014 19:26:42.643328 15158 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:42.643496 15158 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 338.221, Last appended by leader: 221, Current term: 338, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.643682 15158 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:42.643716 15158 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:42.643828 15158 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 338 FOLLOWER]: Advancing to term 339
I1014 19:26:42.703158 14879 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:42.703243 14879 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:42.703287 14879 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 338 FOLLOWER]: Advancing to term 339
I1014 19:26:42.705348 14879 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.705858 14879 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 339 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.706018 14879 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 339 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.706635 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 339 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:42.706635 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 339 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:42.706802 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (vote granted)
I1014 19:26:42.706830 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 339: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:42.707316 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 339 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 339: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:42.708616 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 339 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 339.
I1014 19:26:42.708891 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 339 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.708941 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 339 election: Election decided. Result: candidate won.
I1014 19:26:42.709115 14879 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:42.709172 14879 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 FOLLOWER]: Leader election won for term 339
I1014 19:26:42.709250 14879 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:42.709411 14879 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 338.221, Last appended by leader: 221, Current term: 339, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.710216 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 339 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.710387 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.779204 15142 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:42.779279 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:42.779332 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 339 FOLLOWER]: Advancing to term 340
I1014 19:26:42.780834 15142 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.781203 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 340 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:42.781352 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 340 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.781803 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 340 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:42.781987 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 340 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.782182 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 340 candidate_status { last_received { term: 338 index: 221 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:42.782327 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 339 FOLLOWER]: Advancing to term 340
I1014 19:26:42.782384 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:42.784456 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 340 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 340.
I1014 19:26:42.784763 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 340 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:42.784811 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 340 election: Election decided. Result: candidate won.
I1014 19:26:42.784888 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 340 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 340 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.785079 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:42.785128 15142 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 FOLLOWER]: Leader election won for term 340
I1014 19:26:42.785203 15142 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:42.785387 15142 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 338.221, Last appended by leader: 221, Current term: 340, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.786293 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.786490 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.787796 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 339. Current term is 340. Ops: []
I1014 19:26:42.787889 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 340 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 339. Current term is 340. Ops: []
I1014 19:26:42.788199 14878 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I1014 19:26:42.788586 14878 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 340.222 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 221
I1014 19:26:42.788663 14879 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I1014 19:26:42.788846 14878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 LEADER]: Stepping down as leader of term 339
I1014 19:26:42.788883 14878 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:42.789043 14878 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 339.222, Last appended by leader: 222, Current term: 339, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.789183 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:42.789219 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:42.789294 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 339 FOLLOWER]: Advancing to term 340
I1014 19:26:42.808681 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:42.809504 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:42.811767 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 339 index: 222. Preceding OpId from leader: term: 340 index: 223. (index mismatch)
I1014 19:26:42.811748 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 340 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 338 index: 221. Preceding OpId from leader: term: 340 index: 223. (index mismatch)
I1014 19:26:42.812229 15142 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I1014 19:26:42.812347 15142 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 339.222 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 221
I1014 19:26:42.812631 15142 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I1014 19:26:42.813037 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 221
I1014 19:26:42.813069 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 339.222
I1014 19:26:42.813104 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:42.814138 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.814333 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.814630 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.814813 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.816120 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.816277 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.817109 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.817302 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:42.818123 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.818271 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.818212 15225 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 340 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:42.818384 15225 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:42.818672 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:42.820144 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:42.916395 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 340 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:42.916503 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:42.916564 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 340 FOLLOWER]: Advancing to term 341
I1014 19:26:42.918673 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.919078 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 341 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:42.919219 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 341 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:42.919848 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 341 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:42.920034 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 341 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.920025 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 341 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:42.920205 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 341 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.920370 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 341 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 341 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.920542 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 341 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 341 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:42.920580 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 341 election: Election decided. Result: candidate lost.
I1014 19:26:42.920732 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:42.920789 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Leader election lost for term 341. Reason: could not achieve majority
I1014 19:26:42.921806 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 340. Current term is 341. Ops: []
I1014 19:26:42.922220 15225 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 340.223, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I1014 19:26:42.922420 15142 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Stepping down as leader of term 340
I1014 19:26:42.922459 15142 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:42.922637 15142 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 340.223, Last appended by leader: 223, Current term: 340, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:42.922754 15142 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:42.922775 15142 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:42.922852 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 340 FOLLOWER]: Advancing to term 341
I1014 19:26:42.998427 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:42.998543 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:42.998598 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 340 FOLLOWER]: Advancing to term 341
I1014 19:26:42.999989 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.000355 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 341 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.000486 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 341 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.001031 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 341 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.001096 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 341 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.001197 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:43.001279 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 341: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:43.001673 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 341 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 341: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:43.002714 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 341 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 341.
I1014 19:26:43.002982 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 341 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.003034 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 341 election: Election decided. Result: candidate won.
I1014 19:26:43.003206 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:43.003262 14878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 FOLLOWER]: Leader election won for term 341
I1014 19:26:43.003330 14878 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:43.003480 14878 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 340.223, Last appended by leader: 223, Current term: 341, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.004174 15224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 341 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.004302 15224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.064496 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:43.064589 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:26:43.064646 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 341 FOLLOWER]: Advancing to term 342
I1014 19:26:43.066076 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.066478 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 342 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.066612 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 342 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.067250 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 342 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.067250 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 342 candidate_status { last_received { term: 340 index: 223 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:43.067401 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 341 FOLLOWER]: Advancing to term 342
I1014 19:26:43.067461 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 342 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.067464 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:43.067880 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 342 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 342 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.068994 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 342 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 342.
I1014 19:26:43.069278 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 342 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.069327 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 342 election: Election decided. Result: candidate won.
I1014 19:26:43.069536 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.185s (election complete)
I1014 19:26:43.069594 15164 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 FOLLOWER]: Leader election won for term 342
I1014 19:26:43.069674 15164 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:43.069838 15164 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 340.223, Last appended by leader: 223, Current term: 342, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.070756 15158 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.070909 15158 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.086365 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 341. Current term is 342. Ops: []
I1014 19:26:43.086876 15224 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I1014 19:26:43.087049 15224 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 342.224 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 223
I1014 19:26:43.087108 14878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 LEADER]: Stepping down as leader of term 341
I1014 19:26:43.087153 14878 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:43.087323 14878 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 341.224, Last appended by leader: 224, Current term: 341, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.087502 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:43.087574 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:43.087680 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 341 FOLLOWER]: Advancing to term 342
I1014 19:26:43.157158 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 342 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 340 index: 223. Preceding OpId from leader: term: 342 index: 224. (index mismatch)
I1014 19:26:43.157634 15158 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I1014 19:26:43.158833 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.159015 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.159729 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.159919 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.160208 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 342 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 341 index: 224. Preceding OpId from leader: term: 342 index: 224. (term mismatch)
I1014 19:26:43.160250 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 223
I1014 19:26:43.160269 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 341.224
I1014 19:26:43.160296 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 342 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:43.160259 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.160390 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.160884 15158 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I1014 19:26:43.161979 15224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.162159 15224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.162109 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.162261 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.204730 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:43.205637 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:43.208009 15242 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.208276 15242 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.210091 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:43.211809 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:43.224411  7718 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.508s	user 0.001s	sys 0.000s
I1014 19:26:43.224462  7718 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:43.225126  7718 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:43.225966  7718 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 335 vs 342: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.226174  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.226351  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.226533  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.226692  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.226847  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.227017  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:43.227177  7718 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 342 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.311569 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 342 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:43.311710 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:43.311779 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 342 FOLLOWER]: Advancing to term 343
I1014 19:26:43.313908 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.314399 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 343 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.314599 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 343 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.314913 15142 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 342 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:43.314996 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:43.315048 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 342 FOLLOWER]: Advancing to term 343
I1014 19:26:43.315455 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 343 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.315470 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 343 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.315681 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 343 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.316120 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 343 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 343 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.317023 15142 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.317311 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 343: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:43.317486 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 343 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.317622 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 343 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.318184 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 343 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:43.318184 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 343 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 343: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:43.318434 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 343: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:43.318488 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 343 election: Election decided. Result: candidate lost.
I1014 19:26:43.318725 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:43.318827 14878 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Leader election lost for term 343. Reason: could not achieve majority
I1014 19:26:43.318966 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 343 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 343: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:43.319159 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 343 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.319321 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 343 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.319643 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 343 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 343 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.319707 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 343 election: Election decided. Result: candidate lost.
I1014 19:26:43.319877 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:43.319923 15142 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Leader election lost for term 343. Reason: could not achieve majority
I1014 19:26:43.334128 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 342. Current term is 343. Ops: []
I1014 19:26:43.334625 15242 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 342.225, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I1014 19:26:43.334880 15242 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Stepping down as leader of term 342
I1014 19:26:43.334918 15242 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:43.335095 15242 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 342.225, Last appended by leader: 225, Current term: 342, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.335317 15242 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:43.335359 15242 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:43.335500 15242 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 342 FOLLOWER]: Advancing to term 343
I1014 19:26:43.426097 15142 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:43.426178 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.204s (starting election)
I1014 19:26:43.426231 15142 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 343 FOLLOWER]: Advancing to term 344
I1014 19:26:43.428274 15142 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.428758 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 344 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.428906 15142 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 344 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.429386 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 344 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:43.429572 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 343 FOLLOWER]: Advancing to term 344
I1014 19:26:43.429682 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.222s (vote granted)
I1014 19:26:43.429741 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 344 candidate_status { last_received { term: 342 index: 225 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.429890 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 343 FOLLOWER]: Advancing to term 344
I1014 19:26:43.429960 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:43.431717 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 344.
I1014 19:26:43.431960 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 344 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 344.
I1014 19:26:43.432251 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 344 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.432322 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 344 election: Election decided. Result: candidate won.
I1014 19:26:43.432417 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 344 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.432574 15142 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.223s (election complete)
I1014 19:26:43.432638 15142 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 FOLLOWER]: Leader election won for term 344
I1014 19:26:43.433203 15142 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:43.433418 15142 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 342.225, Last appended by leader: 225, Current term: 344, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.434870 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.435112 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.526118 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:43.527063 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:43.528383 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 342 index: 225. Preceding OpId from leader: term: 344 index: 226. (index mismatch)
I1014 19:26:43.528697 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 344 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 342 index: 225. Preceding OpId from leader: term: 344 index: 226. (index mismatch)
I1014 19:26:43.528914 15273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I1014 19:26:43.529654 15273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I1014 19:26:43.530925 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.531025 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.531170 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.531208 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.532114 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.532312 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.532784 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.532948 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.532976 14878 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.533159 14878 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.533504 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:43.533568 14041 mvcc.cc:203] Tried to move safe_time back from 6306080372851056640 to 6306080372466368512. Current Snapshot: MvccSnapshot[committed={T|T < 6306080372851056640 or (T in {6306080372851056640})}]
I1014 19:26:43.533974 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 344 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.534152 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:43.535401 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:43.729938 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 344 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:43.730028 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:26:43.730083 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 344 FOLLOWER]: Advancing to term 345
I1014 19:26:43.732239 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.732866 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 345 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.733036 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 345 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.734364 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 345 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.734374 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 345 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:43.734591 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 345 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.734602 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 345 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.735064 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 345 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 345 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.735221 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 345 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 345 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.735283 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 345 election: Election decided. Result: candidate lost.
I1014 19:26:43.735507 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:43.735595 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Leader election lost for term 345. Reason: could not achieve majority
I1014 19:26:43.745326 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 344. Current term is 345. Ops: []
I1014 19:26:43.745859 15273 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 344.227, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I1014 19:26:43.746105 15273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 LEADER]: Stepping down as leader of term 344
I1014 19:26:43.746143 15273 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:43.746325 15273 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 344.227, Last appended by leader: 227, Current term: 344, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.746537 15273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:43.746593 15273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:43.746778 15273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 344 FOLLOWER]: Advancing to term 345
I1014 19:26:43.798822 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:43.798909 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:43.798969 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 344 FOLLOWER]: Advancing to term 345
I1014 19:26:43.800949 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.801441 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 345 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.801609 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 345 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.802325 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 345 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.802325 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 345 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.802543 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 345: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:43.802534 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:43.804529 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 345 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 345.
I1014 19:26:43.804893 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 345 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.805007 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 345 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 345: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:43.805097 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 345 election: Election decided. Result: candidate won.
I1014 19:26:43.805330 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:43.805433 14878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 FOLLOWER]: Leader election won for term 345
I1014 19:26:43.806672 14878 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:43.806929 14878 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 344.227, Last appended by leader: 227, Current term: 345, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.808203 15294 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 345 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.808487 15294 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.818722 15296 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:43.820191 15296 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:43.884614 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:43.884711 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.174s (starting election)
I1014 19:26:43.884769 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 345 FOLLOWER]: Advancing to term 346
I1014 19:26:43.886932 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.887461 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 346 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.887621 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 346 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:43.888192 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 346 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.888351 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 345 FOLLOWER]: Advancing to term 346
I1014 19:26:43.888437 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:43.888478 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 346 candidate_status { last_received { term: 344 index: 227 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:43.888662 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 346 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.889149 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 346 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 346 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.890404 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 346 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 346.
I1014 19:26:43.890774 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 346 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.890839 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 346 election: Election decided. Result: candidate won.
I1014 19:26:43.891047 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.207s (election complete)
I1014 19:26:43.891109 15164 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 FOLLOWER]: Leader election won for term 346
I1014 19:26:43.893538 15164 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:43.893746 15164 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 344.227, Last appended by leader: 227, Current term: 346, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.894942 15351 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 346 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.895402 15351 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:43.897943 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 346 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 345. Current term is 346. Ops: []
I1014 19:26:43.898473 15294 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I1014 19:26:43.898749 14878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 LEADER]: Stepping down as leader of term 345
I1014 19:26:43.898802 14878 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:43.898973 14878 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 345.228, Last appended by leader: 228, Current term: 345, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.899158 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:43.899200 14878 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:43.899300 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 345 FOLLOWER]: Advancing to term 346
I1014 19:26:43.976809 14878 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 346 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:43.976919 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:43.976994 14878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 346 FOLLOWER]: Advancing to term 347
I1014 19:26:43.979173 14878 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.979743 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 347 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.979964 14878 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 347 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:43.980527 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 347 candidate_status { last_received { term: 345 index: 228 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:43.980712 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 346 FOLLOWER]: Advancing to term 347
I1014 19:26:43.980835 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:43.982012 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 347 candidate_status { last_received { term: 345 index: 228 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:43.982203 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 347 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.982882 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 347 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 347.
I1014 19:26:43.983283 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 347 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:43.983362 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 347 election: Election decided. Result: candidate won.
I1014 19:26:43.983598 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 347 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 347 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:43.986814 14878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:43.986907 14878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 FOLLOWER]: Leader election won for term 347
I1014 19:26:43.987064 14878 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:43.988879 14878 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 345.228, Last appended by leader: 228, Current term: 347, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:43.990279 15294 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:43.990500 15294 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.004251 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 347 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 346. Current term is 347. Ops: []
I1014 19:26:44.004772 15351 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I1014 19:26:44.005026 15164 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 LEADER]: Stepping down as leader of term 346
I1014 19:26:44.005067 15164 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:44.005234 15164 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 346.228, Last appended by leader: 228, Current term: 346, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.005431 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:44.005470 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:44.005564 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 346 FOLLOWER]: Advancing to term 347
I1014 19:26:44.082777 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 346 index: 228. Preceding OpId from leader: term: 347 index: 229. (index mismatch)
I1014 19:26:44.083294 15294 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 227, Time since last communication: 0.000s
I1014 19:26:44.083477 15294 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 346.228 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 227
I1014 19:26:44.084111 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 227
I1014 19:26:44.084136 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 346.228
I1014 19:26:44.084170 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:44.084688 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.084877 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.086206 15294 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.086387 15294 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.086756 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 347 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 344 index: 227. Preceding OpId from leader: term: 347 index: 229. (index mismatch)
I1014 19:26:44.087414 15294 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 227, Time since last communication: 0.000s
I1014 19:26:44.100715 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.100946 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.101148 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.101321 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.102576 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.102802 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.103399 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.103683 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.103969 15351 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.104130 15351 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.181929 15294 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 347 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.182215 15294 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.361163 15419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 347 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:44.361248 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:44.361321 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 347 FOLLOWER]: Advancing to term 348
I1014 19:26:44.363541 15419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.364043 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 348 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.364202 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 348 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:44.364869 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 348 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:44.365067 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 348 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.365392 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 348 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:44.365605 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 348 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.365577 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 348 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 348 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.366034 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 348 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 348 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.366111 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 348 election: Election decided. Result: candidate lost.
I1014 19:26:44.366500 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:26:44.366591 15419 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Leader election lost for term 348. Reason: could not achieve majority
I1014 19:26:44.378849 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 347. Current term is 348. Ops: []
I1014 19:26:44.379333 14878 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 347.229, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I1014 19:26:44.379632 15421 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 LEADER]: Stepping down as leader of term 347
I1014 19:26:44.379700 15421 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:44.379997 15421 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 347.229, Last appended by leader: 229, Current term: 347, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.380223 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:44.380285 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:44.380456 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 347 FOLLOWER]: Advancing to term 348
I1014 19:26:44.411142 15296 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:44.412128 15296 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:44.413355 15296 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 345 vs 348: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 348 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:44.413671 15296 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 348 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:44.413938 15296 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 348 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:44.415202 15296 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 348 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.445219 15351 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:44.445313 15351 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:44.445376 15351 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 347 FOLLOWER]: Advancing to term 348
I1014 19:26:44.447614 15351 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.448146 15351 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 348 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:44.448307 15351 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 348 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.449059 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 348 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:44.449074 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 348 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:44.449271 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 348: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:44.449245 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:44.449697 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 348 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 348: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:44.451288 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 348 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 348.
I1014 19:26:44.451714 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 348 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.451763 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 348 election: Election decided. Result: candidate won.
I1014 19:26:44.452286 15351 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:44.452354 15351 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 FOLLOWER]: Leader election won for term 348
I1014 19:26:44.452560 15351 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:44.452771 15351 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 347.229, Last appended by leader: 229, Current term: 348, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.453822 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 348 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.454026 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.475651 15419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:44.475780 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.189s (starting election)
I1014 19:26:44.475868 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 348 FOLLOWER]: Advancing to term 349
I1014 19:26:44.478111 15419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.478708 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 349 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.478926 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 349 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:44.479645 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 349 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:44.479833 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 349 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.480288 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 349 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 349 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.480574 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 349 candidate_status { last_received { term: 347 index: 229 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:44.480790 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 348 FOLLOWER]: Advancing to term 349
I1014 19:26:44.480909 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (vote granted)
I1014 19:26:44.483062 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 349 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 349.
I1014 19:26:44.483515 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 349 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.483600 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 349 election: Election decided. Result: candidate won.
I1014 19:26:44.484143 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:44.484233 15419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 FOLLOWER]: Leader election won for term 349
I1014 19:26:44.484472 15419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:44.484702 15419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 347.229, Last appended by leader: 229, Current term: 349, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.486513 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.486712 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.568014 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 348. Current term is 349. Ops: []
I1014 19:26:44.568081 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 349 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 347 index: 229. Preceding OpId from leader: term: 349 index: 230. (index mismatch)
I1014 19:26:44.568841 15164 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I1014 19:26:44.569054 15164 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 349.230 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 229
I1014 19:26:44.569155 15164 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 LEADER]: Stepping down as leader of term 348
I1014 19:26:44.569205 15164 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:44.569383 15164 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 229, Committed index: 229, Last appended: 348.230, Last appended by leader: 230, Current term: 348, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.569603 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:44.569648 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:44.569751 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 348 FOLLOWER]: Advancing to term 349
I1014 19:26:44.572825 15419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I1014 19:26:44.574544 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.574769 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.575027 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 349 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 348. Current term is 349. Ops: []
I1014 19:26:44.576829 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.577091 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.577261 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 348 index: 230. Preceding OpId from leader: term: 349 index: 230. (term mismatch)
I1014 19:26:44.577303 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 229
I1014 19:26:44.577320 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 348.230
I1014 19:26:44.577350 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:44.577952 15142 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 230, Last known committed idx: 229, Time since last communication: 0.000s
I1014 19:26:44.579314 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.579563 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.579767 15351 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.579970 15351 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.580081 15351 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.580231 15351 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.674057 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 349 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.674348 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.679996 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 349 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:44.680091 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:44.680171 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 349 FOLLOWER]: Advancing to term 350
I1014 19:26:44.682225 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 350 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.682740 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 350 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:44.682977 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 350 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:44.683552 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 350 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:44.683562 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 350 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:44.683745 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.683753 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.684188 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 350 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.684315 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 350 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.684466 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 350 election: Election decided. Result: candidate lost.
I1014 19:26:44.684803 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:44.684881 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 350 FOLLOWER]: Leader election lost for term 350. Reason: could not achieve majority
I1014 19:26:44.686285 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 350 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 349. Current term is 350. Ops: []
I1014 19:26:44.686831 15419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 349.230, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I1014 19:26:44.687049 15273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Stepping down as leader of term 349
I1014 19:26:44.687114 15273 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:44.687299 15273 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 349.230, Last appended by leader: 230, Current term: 349, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.687562 15273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:44.687615 15273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:44.687803 15273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 349 FOLLOWER]: Advancing to term 350
I1014 19:26:44.764394 15273 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 350 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:44.764516 15273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:44.764585 15273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 350 FOLLOWER]: Advancing to term 351
I1014 19:26:44.766980 15273 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.767575 15273 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 351 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.768260 15273 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 351 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:44.768354 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 351 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:44.768553 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 350 FOLLOWER]: Advancing to term 351
I1014 19:26:44.768653 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:44.768930 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 351 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:44.769104 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 351 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.769541 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 351 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 351 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.770738 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 351.
I1014 19:26:44.771132 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 351 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.771209 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 351 election: Election decided. Result: candidate won.
I1014 19:26:44.771438 15273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:44.771581 15273 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 FOLLOWER]: Leader election won for term 351
I1014 19:26:44.771796 15273 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:44.772068 15273 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 349.230, Last appended by leader: 230, Current term: 351, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.773133 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.773350 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.775122 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:44.775192 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:44.775236 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 349 FOLLOWER]: Advancing to term 350
I1014 19:26:44.777318 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 350 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:44.777820 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 350 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:44.777990 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 350 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:44.778666 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 350 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:44.778686 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 350 candidate_status { last_received { term: 349 index: 230 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:44.778898 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.778940 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 350. Current term is 351.
W1014 19:26:44.779721 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 350 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 350. Current term is 351.
I1014 19:26:44.779758 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 350 election: Cancelling election due to peer responding with higher term
I1014 19:26:44.779980 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
W1014 19:26:44.780026 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 350 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 350 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:44.780042 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 350 FOLLOWER]: Advancing to term 351
I1014 19:26:44.782096 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 351 FOLLOWER]: Leader election lost for term 350. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 350. Current term is 351.
I1014 19:26:44.888061 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 351 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 349 index: 230. Preceding OpId from leader: term: 351 index: 231. (index mismatch)
I1014 19:26:44.888708 15273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I1014 19:26:44.890475 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.890677 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.891924 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.892194 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.892990 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 349 index: 230. Preceding OpId from leader: term: 351 index: 231. (index mismatch)
I1014 19:26:44.893255 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.893470 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.893839 15273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
W1014 19:26:44.895193 13891 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.360s	user 0.001s	sys 0.000s
I1014 19:26:44.895251 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:44.896270 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:44.903940 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.904012 15676 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.904244 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.904247 15676 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:44.906864 13891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:44.907021 15273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 351 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:44.907469 15273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:44.909020 13891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:45.060356 13891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:45.061640 13891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:45.168349 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 351 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:45.168478 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:45.168543 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 351 FOLLOWER]: Advancing to term 352
I1014 19:26:45.170765 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.171279 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 352 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.172040 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 352 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.172289 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 352 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.172349 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 352 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.172679 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 351. Current term is 352. Ops: []
I1014 19:26:45.172658 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 352 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 352 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.173239 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 352 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.173260 15419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 351.233, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I1014 19:26:45.173460 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 352 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.173544 15419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Stepping down as leader of term 351
I1014 19:26:45.173602 15419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:45.173844 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 352 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 352 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.173910 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 352 election: Election decided. Result: candidate lost.
I1014 19:26:45.173782 15419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 351.233, Last appended by leader: 233, Current term: 351, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.173979 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:45.174032 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:45.174124 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:45.174180 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Leader election lost for term 352. Reason: could not achieve majority
I1014 19:26:45.174178 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 351 FOLLOWER]: Advancing to term 352
W1014 19:26:45.192780 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.981s	user 0.002s	sys 0.000s
I1014 19:26:45.192826 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:45.193784 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:45.194800 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 342 vs 352: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.195066 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.195319 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.195724 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.195986 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.196259 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:45.196571 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 352 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.257318 15676 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:45.257407 15676 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (starting election)
I1014 19:26:45.257517 15676 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 351 FOLLOWER]: Advancing to term 352
I1014 19:26:45.259745 15676 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.260294 15676 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 352 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.260582 15676 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 352 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:45.261234 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 352 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:45.261432 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 352: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:45.261553 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 352 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.261728 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:45.261831 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 352 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 352: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:45.263926 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 352 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 352.
I1014 19:26:45.264744 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 352 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.264834 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 352 election: Election decided. Result: candidate won.
I1014 19:26:45.265079 15676 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:26:45.265159 15676 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 FOLLOWER]: Leader election won for term 352
I1014 19:26:45.265316 15676 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:45.265570 15676 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 351.233, Last appended by leader: 233, Current term: 352, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.266634 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 352 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.266818 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.278779 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:45.281404 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:45.321897 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:45.321964 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:26:45.322010 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 352 FOLLOWER]: Advancing to term 353
I1014 19:26:45.324230 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.324774 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 353 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.324939 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 353 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.325568 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 353 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.325665 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 353 candidate_status { last_received { term: 351 index: 233 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.325783 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 353 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.325824 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 352 FOLLOWER]: Advancing to term 353
I1014 19:26:45.325911 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (vote granted)
I1014 19:26:45.326208 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 353 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 353 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.327956 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 353 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 353.
I1014 19:26:45.328876 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 353 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.328933 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 353 election: Election decided. Result: candidate won.
I1014 19:26:45.329273 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:45.329339 15164 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 FOLLOWER]: Leader election won for term 353
I1014 19:26:45.337520 15164 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:45.337852 15164 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 351.233, Last appended by leader: 233, Current term: 353, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.339499 15739 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 353 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.339958 15739 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.340220 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:45.342223 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:45.343830 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 352. Current term is 353. Ops: []
I1014 19:26:45.344321 15421 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I1014 19:26:45.344558 15421 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 353.234 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 233
I1014 19:26:45.344656 15421 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 LEADER]: Stepping down as leader of term 352
I1014 19:26:45.344700 15421 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:45.344933 15421 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 352.234, Last appended by leader: 234, Current term: 352, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.345126 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:45.345183 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:45.345301 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 352 FOLLOWER]: Advancing to term 353
I1014 19:26:45.423419 15676 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 353 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:45.423535 15676 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:45.423616 15676 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 353 FOLLOWER]: Advancing to term 354
I1014 19:26:45.425822 15676 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.426383 15676 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 354 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.426625 15676 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 354 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:45.427191 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 354 candidate_status { last_received { term: 352 index: 234 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.427345 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 353 FOLLOWER]: Advancing to term 354
I1014 19:26:45.427440 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (vote granted)
I1014 19:26:45.427544 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 354 candidate_status { last_received { term: 352 index: 234 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:45.427745 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 354 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.428191 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 354 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 354 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.429638 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 354.
I1014 19:26:45.430006 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 354 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.430081 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 354 election: Election decided. Result: candidate won.
I1014 19:26:45.430336 15676 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:45.430415 15676 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 FOLLOWER]: Leader election won for term 354
I1014 19:26:45.430660 15676 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:45.430877 15676 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 352.234, Last appended by leader: 234, Current term: 354, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.431908 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.432107 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.451653 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 353. Current term is 354. Ops: []
I1014 19:26:45.452286 15739 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I1014 19:26:45.452544 15164 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Stepping down as leader of term 353
I1014 19:26:45.452586 15164 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:45.452769 15164 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 353.234, Last appended by leader: 234, Current term: 353, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.452966 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:45.453013 15164 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:45.453132 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 353 FOLLOWER]: Advancing to term 354
I1014 19:26:45.517704 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 354 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 353 index: 234. Preceding OpId from leader: term: 354 index: 235. (index mismatch)
I1014 19:26:45.518256 15676 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 233, Time since last communication: 0.000s
I1014 19:26:45.518431 15676 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 353.234 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 233
I1014 19:26:45.519083 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 233
I1014 19:26:45.519109 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 353.234
I1014 19:26:45.519143 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 354 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:45.519670 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.519883 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.521155 15676 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.521356 15676 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.521365 15739 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.521561 15739 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.521515 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.521701 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.521811 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 351 index: 233. Preceding OpId from leader: term: 354 index: 235. (index mismatch)
I1014 19:26:45.522604 15421 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 233, Time since last communication: 0.000s
I1014 19:26:45.526638 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.527030 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.527346 15142 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.527552 15142 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.527632 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.527705 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.602640 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 354 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.602906 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.621666 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 354 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:45.621753 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:45.621803 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 354 FOLLOWER]: Advancing to term 355
I1014 19:26:45.624083 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.624605 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 355 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.624775 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 355 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.625600 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 355 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.625799 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 355 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.625799 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 355 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.625998 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 355 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.626510 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 355 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 355 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.626673 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 355 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 355 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.626737 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 355 election: Election decided. Result: candidate lost.
I1014 19:26:45.626965 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:45.627032 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Leader election lost for term 355. Reason: could not achieve majority
I1014 19:26:45.630208 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 354. Current term is 355. Ops: []
I1014 19:26:45.630784 15676 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 354.235, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I1014 19:26:45.631083 15676 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 LEADER]: Stepping down as leader of term 354
I1014 19:26:45.631151 15676 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:45.631340 15676 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 354.235, Last appended by leader: 235, Current term: 354, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.631569 15676 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:45.631625 15676 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:45.631775 15676 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 354 FOLLOWER]: Advancing to term 355
I1014 19:26:45.704974 15419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:45.705080 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:45.705155 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 354 FOLLOWER]: Advancing to term 355
I1014 19:26:45.707401 15419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.707918 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 355 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.708101 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 355 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:45.708940 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 355 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.708943 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 355 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:45.709112 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (vote granted)
I1014 19:26:45.709183 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 355: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:45.709730 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 355 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 355: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:45.711199 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 355 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 355.
I1014 19:26:45.711668 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 355 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.711745 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 355 election: Election decided. Result: candidate won.
I1014 19:26:45.712405 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:45.712576 15419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 FOLLOWER]: Leader election won for term 355
I1014 19:26:45.712822 15419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:45.713093 15419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 354.235, Last appended by leader: 235, Current term: 355, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.715145 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 355 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.715355 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.716270 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:45.719681 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:45.753026 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:45.753139 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.208s (starting election)
I1014 19:26:45.753204 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 355 FOLLOWER]: Advancing to term 356
I1014 19:26:45.755554 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.756059 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 356 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:45.756289 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 356 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.756945 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 356 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.757078 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 356 candidate_status { last_received { term: 354 index: 235 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:45.757160 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 355 FOLLOWER]: Advancing to term 356
I1014 19:26:45.757261 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (vote granted)
I1014 19:26:45.757303 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 356 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.758024 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 356 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 356 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.759366 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 356.
I1014 19:26:45.759737 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 356 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.759820 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 356 election: Election decided. Result: candidate won.
I1014 19:26:45.760174 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:45.760272 15164 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 FOLLOWER]: Leader election won for term 356
I1014 19:26:45.760432 15164 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:45.760658 15164 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 354.235, Last appended by leader: 235, Current term: 356, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.762104 15739 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.762300 15739 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.798414 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 355. Current term is 356. Ops: []
I1014 19:26:45.799015 15419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I1014 19:26:45.799290 15419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 LEADER]: Stepping down as leader of term 355
I1014 19:26:45.799350 15419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:45.799607 15419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 355.236, Last appended by leader: 236, Current term: 355, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.799805 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:45.799871 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:45.799994 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 355 FOLLOWER]: Advancing to term 356
I1014 19:26:45.846554 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 354 index: 235. Preceding OpId from leader: term: 356 index: 236. (index mismatch)
I1014 19:26:45.847241 15164 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I1014 19:26:45.848734 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.848956 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.850428 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.850693 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.851692 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 356 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 355 index: 236. Preceding OpId from leader: term: 356 index: 236. (term mismatch)
I1014 19:26:45.851749 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 235
I1014 19:26:45.851783 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 355.236
I1014 19:26:45.851824 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 356 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:45.852254 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.852458 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.853161 15739 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I1014 19:26:45.855130 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.855345 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.855489 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.855680 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:45.954530 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 356 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:45.954636 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:45.954713 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 356 FOLLOWER]: Advancing to term 357
I1014 19:26:45.956041 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 356 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.956284 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:45.956914 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.957425 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 357 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:45.958155 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 356. Current term is 357. Ops: []
I1014 19:26:45.958559 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 357 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:45.958992 15739 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 356.236, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I1014 19:26:45.959118 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 357 candidate_status { last_received { term: 356 index: 236 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:45.959223 15739 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 LEADER]: Stepping down as leader of term 356
I1014 19:26:45.959290 15739 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:45.959324 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 357 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.959506 15739 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 356.236, Last appended by leader: 236, Current term: 356, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.959703 15739 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:45.959764 15739 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:45.959887 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 357 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 357 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:45.959954 15739 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 356 FOLLOWER]: Advancing to term 357
I1014 19:26:45.962283 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 357 candidate_status { last_received { term: 356 index: 236 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:45.962477 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (vote granted)
I1014 19:26:45.964537 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 357.
I1014 19:26:45.964900 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 357 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:45.964980 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 357 election: Election decided. Result: candidate won.
I1014 19:26:45.965242 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:26:45.965313 15762 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 FOLLOWER]: Leader election won for term 357
I1014 19:26:45.965409 15762 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:45.965625 15762 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 356.236, Last appended by leader: 236, Current term: 357, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:45.966605 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:45.966812 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.049702 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:46.049798 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:46.049868 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 356 FOLLOWER]: Advancing to term 357
I1014 19:26:46.051967 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 357 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.052506 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 357 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.052945 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 357 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.053040 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 357 candidate_status { last_received { term: 356 index: 236 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.053237 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 357 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.053781 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 357 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 357 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.057348 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 357 candidate_status { last_received { term: 356 index: 236 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.057611 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 357: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:46.057971 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 357 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 357: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:46.058063 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 357 election: Election decided. Result: candidate lost.
I1014 19:26:46.058353 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:46.058439 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 357 FOLLOWER]: Leader election lost for term 357. Reason: could not achieve majority
I1014 19:26:46.064375 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 356 index: 236. Preceding OpId from leader: term: 357 index: 237. (index mismatch)
I1014 19:26:46.064952 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I1014 19:26:46.065963 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 357 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 356 index: 236. Preceding OpId from leader: term: 357 index: 237. (index mismatch)
I1014 19:26:46.066267 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.066534 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.066447 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I1014 19:26:46.067576 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.067795 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.068089 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.068321 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.069283 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.069504 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.069893 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.070083 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.070629 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 357 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.070818 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.170104 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:46.170204 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:46.170248 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 357 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:46.170284 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 357 FOLLOWER]: Advancing to term 358
I1014 19:26:46.170333 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:26:46.170393 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 357 FOLLOWER]: Advancing to term 358
I1014 19:26:46.172536 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.172787 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 358 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.173066 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 358 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.173290 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 358 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.173450 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 358 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.173491 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 358 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.174084 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 358 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.174098 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 358 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.174175 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 358 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.174289 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 358 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.174310 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 358 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 358: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:46.174360 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 358: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:46.174798 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 358 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 358 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.174847 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 358 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 358: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:46.174914 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 358 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 358 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 358: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:46.175002 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 358 election: Election decided. Result: candidate lost.
I1014 19:26:46.175026 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 358 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.175179 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 358 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.175241 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:46.175318 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Leader election lost for term 358. Reason: could not achieve majority
I1014 19:26:46.175546 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 358 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 358 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.175628 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 358 election: Election decided. Result: candidate lost.
I1014 19:26:46.175878 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:26:46.175979 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 358 FOLLOWER]: Leader election lost for term 358. Reason: could not achieve majority
I1014 19:26:46.179256 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 357. Current term is 358. Ops: []
I1014 19:26:46.179776 15762 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 357.237, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I1014 19:26:46.180025 15419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Stepping down as leader of term 357
I1014 19:26:46.180085 15419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:46.180315 15419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 357.237, Last appended by leader: 237, Current term: 357, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.180546 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:46.180609 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:46.180750 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 357 FOLLOWER]: Advancing to term 358
I1014 19:26:46.266896 15419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 358 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:46.267002 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:46.267094 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 358 FOLLOWER]: Advancing to term 359
I1014 19:26:46.269351 15419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.269914 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 359 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.270174 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 359 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.270781 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 359 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.270951 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 358 FOLLOWER]: Advancing to term 359
I1014 19:26:46.271039 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.201s (vote granted)
I1014 19:26:46.271041 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 359 candidate_status { last_received { term: 357 index: 237 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.271231 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 358 FOLLOWER]: Advancing to term 359
I1014 19:26:46.271322 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:26:46.273057 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 359.
I1014 19:26:46.273530 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 359 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.273597 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 359 election: Election decided. Result: candidate won.
I1014 19:26:46.273720 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 359 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 359.
I1014 19:26:46.273815 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:46.273892 15419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 FOLLOWER]: Leader election won for term 359
I1014 19:26:46.274049 15419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:46.274314 15419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 357.237, Last appended by leader: 237, Current term: 359, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.275414 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.275804 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 359 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.275859 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.372098 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 359 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 357 index: 237. Preceding OpId from leader: term: 359 index: 238. (index mismatch)
I1014 19:26:46.372603 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I1014 19:26:46.373891 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.374115 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.375550 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.375780 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.376662 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 357 index: 237. Preceding OpId from leader: term: 359 index: 238. (index mismatch)
I1014 19:26:46.376694 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.376967 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.377410 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I1014 19:26:46.379170 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.379443 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.379170 15992 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.380904 15992 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.476548 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 359 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:46.476636 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:46.476708 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 359 FOLLOWER]: Advancing to term 360
I1014 19:26:46.476671 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 359 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.476925 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.479073 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.484886 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 360 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.485150 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 360 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.486158 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 360 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.486196 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 360 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.486385 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 360 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.486423 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 360 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.486871 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 360 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 360 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.487046 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 360 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 360 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.487131 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 360 election: Election decided. Result: candidate lost.
I1014 19:26:46.492578 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:46.492709 15164 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Leader election lost for term 360. Reason: could not achieve majority
I1014 19:26:46.499137 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 359. Current term is 360. Ops: []
I1014 19:26:46.499634 15419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 359.238, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I1014 19:26:46.499894 15762 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 LEADER]: Stepping down as leader of term 359
I1014 19:26:46.499958 15762 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:46.500159 15762 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 359.238, Last appended by leader: 238, Current term: 359, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.500362 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:46.500428 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:46.500608 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 359 FOLLOWER]: Advancing to term 360
I1014 19:26:46.578788 15992 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:46.578886 15992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:46.578955 15992 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 359 FOLLOWER]: Advancing to term 360
I1014 19:26:46.580984 15992 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.581511 15992 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 360 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.581707 15992 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 360 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.582226 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 360 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.582420 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (vote granted)
I1014 19:26:46.582406 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 360 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.582653 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 360: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:46.583083 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 360 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 360: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:46.584584 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 360 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 360.
I1014 19:26:46.585005 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 360 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.585078 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 360 election: Election decided. Result: candidate won.
I1014 19:26:46.585352 15992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:46.585445 15992 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 FOLLOWER]: Leader election won for term 360
I1014 19:26:46.585546 15992 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:46.585788 15992 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 359.238, Last appended by leader: 238, Current term: 360, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.586838 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 360 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.587049 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.603636 15164 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:46.603746 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.174s (starting election)
I1014 19:26:46.603822 15164 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 360 FOLLOWER]: Advancing to term 361
I1014 19:26:46.606245 15164 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.606858 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 361 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.607103 15164 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 361 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.607770 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 361 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.607990 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 361 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.607883 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 361 candidate_status { last_received { term: 359 index: 238 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.608073 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 360 FOLLOWER]: Advancing to term 361
I1014 19:26:46.608171 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:46.608479 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 361 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 361 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.610417 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 361 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 361.
I1014 19:26:46.611014 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 361 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.611083 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 361 election: Election decided. Result: candidate won.
I1014 19:26:46.611390 15164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.164s (election complete)
I1014 19:26:46.611497 15164 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 FOLLOWER]: Leader election won for term 361
W1014 19:26:46.614074 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.272s	user 0.001s	sys 0.000s
I1014 19:26:46.614136 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:46.615120 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:46.627110 15164 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:46.627506 15164 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 359.238, Last appended by leader: 238, Current term: 361, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.630579 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.630959 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.643656 28825 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 LEADER]: Stepping down as leader of term 360
I1014 19:26:46.643734 28825 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:46.643756 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 361 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 359 index: 238. Preceding OpId from leader: term: 361 index: 240. (index mismatch)
I1014 19:26:46.643950 28825 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 360.239, Last appended by leader: 239, Current term: 360, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.644217 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:46.644268 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:46.644368 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 360 FOLLOWER]: Advancing to term 361
I1014 19:26:46.644358 15164 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I1014 19:26:46.646320 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.646667 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.647413 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 360 index: 239. Preceding OpId from leader: term: 361 index: 240. (index mismatch)
I1014 19:26:46.648881 16149 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I1014 19:26:46.650279 16149 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 360.239 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 238
I1014 19:26:46.651361 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 238
I1014 19:26:46.651404 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 360.239
I1014 19:26:46.651465 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:46.658820 15164 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.659179 15164 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.659524 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.659850 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.661530 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:46.663142 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.663383 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.663579 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:46.663642 15992 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.663836 15992 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.735188 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.735453 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.749236 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 361 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:46.749356 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:46.749445 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 361 FOLLOWER]: Advancing to term 362
I1014 19:26:46.751819 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.752353 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 362 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.753195 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 362 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.753809 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 362 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.754006 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 362 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.754355 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 362 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 362 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.754650 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 362 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.754848 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 362 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.755156 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 362 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 362 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.755226 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 362 election: Election decided. Result: candidate lost.
I1014 19:26:46.755470 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:46.755576 15762 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Leader election lost for term 362. Reason: could not achieve majority
I1014 19:26:46.772449 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 361. Current term is 362. Ops: []
I1014 19:26:46.773031 16149 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 361.240, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I1014 19:26:46.773326 16149 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 LEADER]: Stepping down as leader of term 361
I1014 19:26:46.773396 16149 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:46.773631 16149 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 361.240, Last appended by leader: 240, Current term: 361, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.773854 16149 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:46.773906 16149 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:46.774065 16149 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 361 FOLLOWER]: Advancing to term 362
W1014 19:26:46.797139 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.504s	user 0.002s	sys 0.000s
I1014 19:26:46.797214 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:46.798246 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:46.799553 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 352 vs 361: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:46.799870 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:46.800140 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:46.800397 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:46.800668 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 361 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.835988 15992 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:46.836076 15992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:46.836160 15992 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 361 FOLLOWER]: Advancing to term 362
I1014 19:26:46.837641 15992 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.838119 15992 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 362 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:46.838337 15992 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 362 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.839002 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 362 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:46.839207 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 362: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:46.839205 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 362 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.839390 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:46.841559 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 362 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 362.
I1014 19:26:46.842139 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 362 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.842247 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 362 election: Election decided. Result: candidate won.
I1014 19:26:46.842353 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 362 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 362: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:46.842576 15992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:46.842659 15992 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 FOLLOWER]: Leader election won for term 362
I1014 19:26:46.842780 15992 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:46.843006 15992 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 361.240, Last appended by leader: 240, Current term: 362, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.845921 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 362 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.846161 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.846457 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:46.848335 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:46.899966 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:46.900076 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:46.900146 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 362 FOLLOWER]: Advancing to term 363
I1014 19:26:46.901985 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.902389 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 363 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:46.902645 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 363 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.903298 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 363 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:46.903499 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 362 FOLLOWER]: Advancing to term 363
I1014 19:26:46.903599 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (vote granted)
I1014 19:26:46.904851 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 363 candidate_status { last_received { term: 361 index: 240 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:46.905050 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 363 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.905409 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 363 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 363 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:46.905807 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 363 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 363.
I1014 19:26:46.906601 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 363 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:46.906672 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 363 election: Election decided. Result: candidate won.
I1014 19:26:46.907122 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:46.907213 15762 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 FOLLOWER]: Leader election won for term 363
I1014 19:26:46.907354 15762 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:46.907655 15762 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 361.240, Last appended by leader: 240, Current term: 363, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.908933 15419 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.909142 15419 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.947127 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 362. Current term is 363. Ops: []
I1014 19:26:46.947724 15992 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I1014 19:26:46.947932 15992 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 363.241 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 240
I1014 19:26:46.948007 15421 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 LEADER]: Stepping down as leader of term 362
I1014 19:26:46.948068 15421 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:46.948253 15421 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 362.241, Last appended by leader: 241, Current term: 362, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:46.948400 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:46.948449 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:46.948567 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 362 FOLLOWER]: Advancing to term 363
I1014 19:26:46.985611 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 363 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 361 index: 240. Preceding OpId from leader: term: 363 index: 241. (index mismatch)
I1014 19:26:46.986243 15419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I1014 19:26:46.987483 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.987741 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.988956 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.989147 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:46.989210 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 362 index: 241. Preceding OpId from leader: term: 363 index: 241. (term mismatch)
I1014 19:26:46.989279 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 240
I1014 19:26:46.989318 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 362.241
I1014 19:26:46.989311 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.989567 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.995560 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:46.996353 15419 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s
I1014 19:26:46.997957 15992 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.998158 15992 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:46.998236 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 363 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:46.998412 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.090020 16143 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 363 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:47.090116 16143 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:47.090196 16143 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 363 FOLLOWER]: Advancing to term 364
I1014 19:26:47.092581 16143 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.093341 16143 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 364 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.093634 16143 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 364 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.094552 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 364 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.094545 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 364 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.094739 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.094743 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.095162 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 364 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.095312 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 364 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.095386 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 364 election: Election decided. Result: candidate lost.
I1014 19:26:47.095680 16143 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:47.095765 16143 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Leader election lost for term 364. Reason: could not achieve majority
I1014 19:26:47.099375 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:47.099449 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:47.099532 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 363 FOLLOWER]: Advancing to term 364
I1014 19:26:47.101672 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 364 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.102162 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 364 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.102313 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 364 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.103346 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 364 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.103606 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 364: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:47.103911 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 364 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.104002 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 364 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 364: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:47.104085 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.104717 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 364 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 364 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.104790 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 364 election: Election decided. Result: candidate lost.
I1014 19:26:47.105168 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 363. Current term is 364. Ops: []
I1014 19:26:47.105159 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:47.105219 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 364 FOLLOWER]: Leader election lost for term 364. Reason: could not achieve majority
I1014 19:26:47.105710 15419 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 363.241, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I1014 19:26:47.105932 15419 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Stepping down as leader of term 363
I1014 19:26:47.105969 15419 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:47.106137 15419 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 363.241, Last appended by leader: 241, Current term: 363, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.106312 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:47.106353 15419 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:47.106509 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 363 FOLLOWER]: Advancing to term 364
W1014 19:26:47.144742 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.425s	user 0.002s	sys 0.000s
I1014 19:26:47.144803 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:47.145730 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:47.147379 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 355 vs 364: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.149013 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:47.149917 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:47.151036 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 362 vs 364: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.152038 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.152284 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.152696 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.152940 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.153185 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.153435 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.153702 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:47.153949 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 364 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.224143 15419 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 364 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:47.224225 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:47.224263 15419 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 364 FOLLOWER]: Advancing to term 365
I1014 19:26:47.225723 15419 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.226189 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 365 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.226349 15419 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 365 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.226892 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 364 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:47.226958 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:47.227010 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 364 FOLLOWER]: Advancing to term 365
I1014 19:26:47.226999 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 365 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.227002 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 365 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.227177 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 364 FOLLOWER]: Advancing to term 365
I1014 19:26:47.227283 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.174s (vote granted)
I1014 19:26:47.228988 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.229281 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 365: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:47.229432 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 365 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.229391 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 365.
I1014 19:26:47.229595 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 365 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.229769 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 365 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 365: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:47.230046 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 365 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.230193 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 365: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.230440 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 365 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.230520 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 365 election: Election decided. Result: candidate won.
I1014 19:26:47.230475 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 365 candidate_status { last_received { term: 363 index: 241 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.230661 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 365: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.230960 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 365 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 365: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.231089 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 365 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 365: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.231158 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 365 election: Election decided. Result: candidate lost.
I1014 19:26:47.231438 15419 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:47.231514 15419 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 FOLLOWER]: Leader election won for term 365
I1014 19:26:47.231595 15419 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:47.231765 15419 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 363.241, Last appended by leader: 241, Current term: 365, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.232981 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:47.233033 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Leader election lost for term 365. Reason: could not achieve majority
I1014 19:26:47.232971 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.233158 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.233331 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:47.235078 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:47.309211 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 363 index: 241. Preceding OpId from leader: term: 365 index: 242. (index mismatch)
I1014 19:26:47.310132 16273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I1014 19:26:47.311576 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.311746 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.312325 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.312618 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.313248 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 363 index: 241. Preceding OpId from leader: term: 365 index: 242. (index mismatch)
I1014 19:26:47.313257 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.313421 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.313804 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s
I1014 19:26:47.315526 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.315526 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 365 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.315865 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.315867 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.413882 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:47.413969 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:47.414023 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 365 FOLLOWER]: Advancing to term 366
I1014 19:26:47.416168 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.416501 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 365. Current term is 366. Ops: []
I1014 19:26:47.416726 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 366 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.417196 15762 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 365.242, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I1014 19:26:47.417459 15762 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 LEADER]: Stepping down as leader of term 365
I1014 19:26:47.417497 15762 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:47.417551 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 366 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.417691 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:47.417675 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 366 candidate_status { last_received { term: 365 index: 242 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.417790 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:47.417707 15762 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 365.242, Last appended by leader: 242, Current term: 365, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.417879 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:47.417918 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:47.417901 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 365 FOLLOWER]: Advancing to term 366
I1014 19:26:47.418051 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 365 FOLLOWER]: Advancing to term 366
I1014 19:26:47.418136 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 366 candidate_status { last_received { term: 365 index: 242 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.420209 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (vote granted)
I1014 19:26:47.420228 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.420677 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 366: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:47.420815 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 366 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.421041 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 366 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.421881 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 366 candidate_status { last_received { term: 365 index: 242 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.422199 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 366.
I1014 19:26:47.422168 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 366 candidate_status { last_received { term: 365 index: 242 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.422355 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 366: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:47.422641 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 366 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:47.423100 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 366 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 366: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:47.423429 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 366 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 366: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:47.423900 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 366 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.423949 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 366 election: Election decided. Result: candidate won.
I1014 19:26:47.424156 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:47.424221 15421 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Leader election won for term 366
I1014 19:26:47.424307 15421 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:47.424301 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 366 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 366 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:47.424376 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 366 election: Election decided. Result: candidate lost.
I1014 19:26:47.424595 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:47.424522 15421 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 365.242, Last appended by leader: 242, Current term: 366, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.424698 16354 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Leader election lost for term 366. Reason: could not achieve majority
I1014 19:26:47.425858 15992 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.426034 15992 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.426211 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:47.428056 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:47.502383 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 365 index: 242. Preceding OpId from leader: term: 366 index: 243. (index mismatch)
I1014 19:26:47.503017 15992 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I1014 19:26:47.504587 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.504792 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.505437 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.505650 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.506001 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 365 index: 242. Preceding OpId from leader: term: 366 index: 243. (index mismatch)
I1014 19:26:47.506129 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.506325 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.506538 15992 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I1014 19:26:47.507988 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.508205 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.508322 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.508472 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.608491 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:47.608608 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:47.608654 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 366 FOLLOWER]: Advancing to term 367
I1014 19:26:47.610647 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.611142 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 367 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.611318 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 367 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.612192 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 367 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.612437 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 367 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.612541 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 367 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.612717 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 367 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.612874 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 367 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 367 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.613066 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 367 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 367 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.613126 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 367 election: Election decided. Result: candidate lost.
I1014 19:26:47.613301 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:47.613356 15762 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Leader election lost for term 367. Reason: could not achieve majority
I1014 19:26:47.619355 15992 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 366 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.619662 15992 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.620198 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 366. Current term is 367. Ops: []
I1014 19:26:47.620936 15992 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 366.243, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I1014 19:26:47.631649 15421 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 LEADER]: Stepping down as leader of term 366
I1014 19:26:47.631774 15421 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:47.632164 15421 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 366.243, Last appended by leader: 243, Current term: 366, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.632459 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:47.632529 15421 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:47.632700 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 366 FOLLOWER]: Advancing to term 367
I1014 19:26:47.704488 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:47.704594 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:47.704679 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 366 FOLLOWER]: Advancing to term 367
I1014 19:26:47.706933 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.707497 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 367 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.707727 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 367 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.708283 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 367 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.708500 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 367: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.708396 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 367 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.708600 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (vote granted)
I1014 19:26:47.710551 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 367 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 367.
I1014 19:26:47.711486 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 367 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.711591 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 367 election: Election decided. Result: candidate won.
I1014 19:26:47.711858 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:47.711915 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 367 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 367: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:47.711992 16354 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 FOLLOWER]: Leader election won for term 367
I1014 19:26:47.712206 16354 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:47.712447 16354 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 366.243, Last appended by leader: 243, Current term: 367, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.713884 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 367 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.714083 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.731199 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:47.731284 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:47.731339 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 367 FOLLOWER]: Advancing to term 368
I1014 19:26:47.733456 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.733986 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 368 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.734141 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 368 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.734807 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 368 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.735008 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 368 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.735090 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 368 candidate_status { last_received { term: 366 index: 243 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:47.735239 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 367 FOLLOWER]: Advancing to term 368
I1014 19:26:47.735312 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (vote granted)
I1014 19:26:47.735355 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 368 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 368 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.737222 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 368 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 368.
I1014 19:26:47.737561 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 368 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:47.737612 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 368 election: Election decided. Result: candidate won.
I1014 19:26:47.737934 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:47.737993 15762 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 FOLLOWER]: Leader election won for term 368
I1014 19:26:47.738205 15762 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:47.738412 15762 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 366.243, Last appended by leader: 243, Current term: 368, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.739380 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.739578 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.817646 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 368 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 367. Current term is 368. Ops: []
I1014 19:26:47.818230 16143 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I1014 19:26:47.818528 16354 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 LEADER]: Stepping down as leader of term 367
I1014 19:26:47.818608 16354 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:47.818914 16354 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 243, Committed index: 243, Last appended: 367.244, Last appended by leader: 244, Current term: 367, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.819133 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:47.819208 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:47.819347 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 367 FOLLOWER]: Advancing to term 368
I1014 19:26:47.826934 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 367 index: 244. Preceding OpId from leader: term: 368 index: 244. (term mismatch)
I1014 19:26:47.827013 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 243
I1014 19:26:47.827059 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 367.244
I1014 19:26:47.827116 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:47.827989 15762 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I1014 19:26:47.829309 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.829634 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.830665 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.830852 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.831195 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 368 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 366 index: 243. Preceding OpId from leader: term: 368 index: 244. (index mismatch)
I1014 19:26:47.831562 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.831779 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.831728 16273 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 243, Time since last communication: 0.000s
I1014 19:26:47.833000 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.833199 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.833032 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.833286 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:47.933145 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 368 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:47.933228 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:47.933285 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 368 FOLLOWER]: Advancing to term 369
I1014 19:26:47.935362 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.935894 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 369 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:47.936065 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 369 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:47.936699 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 369 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:47.936904 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 369 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.937191 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 369 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:47.937364 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 369 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.937697 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 369 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 369 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.937960 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 369 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 369 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:47.938014 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 369 election: Election decided. Result: candidate lost.
I1014 19:26:47.938190 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:47.938249 15421 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Leader election lost for term 369. Reason: could not achieve majority
I1014 19:26:47.955696 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 368 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:47.955935 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:47.956212 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 368. Current term is 369. Ops: []
I1014 19:26:47.957787 15762 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 368.244, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I1014 19:26:47.957974 15762 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 LEADER]: Stepping down as leader of term 368
I1014 19:26:47.958014 15762 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:47.958161 15762 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 368.244, Last appended by leader: 244, Current term: 368, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:47.958349 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:47.958382 15762 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:47.958513 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 368 FOLLOWER]: Advancing to term 369
I1014 19:26:48.009052 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:48.009945 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:48.011853 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 366 vs 369: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 369 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.012306 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:48.012395 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:48.012480 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 368 FOLLOWER]: Advancing to term 369
W1014 19:26:48.012557 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 369 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.012784 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 369 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.014008 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.014372 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 369 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.014537 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 369 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.015053 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 369 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.015210 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
I1014 19:26:48.015396 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 369 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.015588 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 369: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.015951 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 369 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 369: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.017132 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 369 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 369.
I1014 19:26:48.017452 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 369 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.017524 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 369 election: Election decided. Result: candidate won.
I1014 19:26:48.017695 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:48.017753 16354 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 FOLLOWER]: Leader election won for term 369
I1014 19:26:48.017853 16354 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.018080 16354 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 368.244, Last appended by leader: 244, Current term: 369, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.024626 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 369 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.024873 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.088102 15421 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:48.088191 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:48.088253 15421 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 369 FOLLOWER]: Advancing to term 370
I1014 19:26:48.090385 15421 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.090962 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 370 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.091140 15421 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 370 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.091785 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 370 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.091951 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 369 FOLLOWER]: Advancing to term 370
I1014 19:26:48.092044 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (vote granted)
I1014 19:26:48.092018 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 370 candidate_status { last_received { term: 368 index: 244 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.092224 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 370 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.092706 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 370 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 370 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.093907 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 370.
I1014 19:26:48.094254 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 370 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.094310 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 370 election: Election decided. Result: candidate won.
I1014 19:26:48.094532 15421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.157s (election complete)
I1014 19:26:48.094594 15421 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 FOLLOWER]: Leader election won for term 370
I1014 19:26:48.094749 15421 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:48.094936 15421 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 368.244, Last appended by leader: 244, Current term: 370, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.095947 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.096128 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.096328 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:48.098147 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:48.101615 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 369. Current term is 370. Ops: []
I1014 19:26:48.102180 16143 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I1014 19:26:48.102389 16143 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 370.245 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 244
I1014 19:26:48.102458 16354 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 LEADER]: Stepping down as leader of term 369
I1014 19:26:48.102510 16354 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.102723 16354 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 369.245, Last appended by leader: 245, Current term: 369, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.102916 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:48.102950 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:48.103060 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 369 FOLLOWER]: Advancing to term 370
I1014 19:26:48.174841 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 370 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 369 index: 245. Preceding OpId from leader: term: 370 index: 245. (term mismatch)
I1014 19:26:48.174918 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 244
I1014 19:26:48.174942 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 369.245
I1014 19:26:48.175002 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 370 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:48.175684 15421 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I1014 19:26:48.177258 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.177480 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.178382 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.178593 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.179011 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 368 index: 244. Preceding OpId from leader: term: 370 index: 245. (index mismatch)
I1014 19:26:48.179774 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.179957 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.180049 16480 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I1014 19:26:48.181392 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.181607 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.181721 16273 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.181877 16273 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.260632 15421 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.260877 15421 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:48.277374 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.614s	user 0.001s	sys 0.000s
I1014 19:26:48.277453 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:48.278280 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:48.279803 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 361 vs 370: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.280398 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.280882 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.281224 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.281597 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 370 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.281692 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 370 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:48.281801 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:48.281853 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 370 FOLLOWER]: Advancing to term 371
I1014 19:26:48.283938 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.284262 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 370. Current term is 371. Ops: []
I1014 19:26:48.284466 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 371 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.284602 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 371 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.285356 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 371 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
W1014 19:26:48.285457 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 371 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.285576 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 371 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.285619 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 371 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.285815 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 371 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.285845 16706 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 370.245, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I1014 19:26:48.286061 16480 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Stepping down as leader of term 370
I1014 19:26:48.286101 16480 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
W1014 19:26:48.286157 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 371 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.286193 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 371 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 371 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.286387 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 371 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 371 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.286283 16480 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 370.245, Last appended by leader: 245, Current term: 370, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.286442 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 371 election: Election decided. Result: candidate lost.
I1014 19:26:48.286458 16480 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:48.286495 16480 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:48.286645 16480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 370 FOLLOWER]: Advancing to term 371
I1014 19:26:48.286749 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:48.286799 16354 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Leader election lost for term 371. Reason: could not achieve majority
W1014 19:26:48.287178 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 371 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.361693 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:48.361780 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:48.361834 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 370 FOLLOWER]: Advancing to term 371
I1014 19:26:48.363272 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.363692 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 371 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.363852 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 371 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.364413 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 371 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.364603 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:48.364560 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 371 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.364761 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 371: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:48.365263 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 371 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 371: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:48.366675 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 371 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 371.
I1014 19:26:48.367017 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 371 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.367069 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 371 election: Election decided. Result: candidate won.
I1014 19:26:48.367244 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:26:48.367300 15762 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 FOLLOWER]: Leader election won for term 371
I1014 19:26:48.367377 15762 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:48.367563 15762 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 370.245, Last appended by leader: 245, Current term: 371, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.368336 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 371 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.368535 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.393494 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:48.393584 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.209s (starting election)
I1014 19:26:48.393649 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 371 FOLLOWER]: Advancing to term 372
I1014 19:26:48.395561 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.396090 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 372 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.396260 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 372 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.396976 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 372 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.397147 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 371 FOLLOWER]: Advancing to term 372
I1014 19:26:48.397217 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (vote granted)
I1014 19:26:48.397187 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 372 candidate_status { last_received { term: 370 index: 245 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.397373 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 372 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.399197 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 372 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 372.
I1014 19:26:48.399600 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 372 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.399667 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 372 election: Election decided. Result: candidate won.
I1014 19:26:48.403687 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:48.403678 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 372 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 372 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.403771 16354 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 FOLLOWER]: Leader election won for term 372
I1014 19:26:48.403878 16354 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.404094 16354 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 370.245, Last appended by leader: 245, Current term: 372, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.405398 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.405634 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.405818 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:48.407722 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:48.476430 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 371. Current term is 372. Ops: []
I1014 19:26:48.477171 16273 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I1014 19:26:48.477319 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 372 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 371. Current term is 372. Ops: []
I1014 19:26:48.477360 16273 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 372.246 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 245
I1014 19:26:48.477457 16273 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 LEADER]: Stepping down as leader of term 371
I1014 19:26:48.477488 16273 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:48.477639 16273 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 371.246, Last appended by leader: 246, Current term: 371, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.477811 16273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:48.477843 16273 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:48.477946 16273 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 371 FOLLOWER]: Advancing to term 372
I1014 19:26:48.484481 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 372 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 371 index: 246. Preceding OpId from leader: term: 372 index: 246. (term mismatch)
I1014 19:26:48.484556 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 245
I1014 19:26:48.484575 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 371.246
I1014 19:26:48.484609 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 372 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:48.485149 16143 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I1014 19:26:48.486475 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.486688 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.487556 16143 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.487745 16143 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.488389 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.488576 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.488744 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 372 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 370 index: 245. Preceding OpId from leader: term: 372 index: 246. (index mismatch)
I1014 19:26:48.489135 16810 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s
I1014 19:26:48.490474 16706 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.490734 16706 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.490630 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.490777 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.588891 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 372 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:48.588981 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:48.589035 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 372 FOLLOWER]: Advancing to term 373
I1014 19:26:48.590965 16480 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 372 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:48.591038 16480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:48.591090 16480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 372 FOLLOWER]: Advancing to term 373
I1014 19:26:48.591095 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.591603 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 373 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.591773 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 373 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.592597 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 373 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.592824 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 373 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.592787 16480 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.593181 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 373 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 373 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.593227 16480 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 373 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.593353 16480 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 373 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.593598 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 373 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.593741 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 373: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.593783 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 373 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.593967 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 373: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:48.594067 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 373 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 373: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.594187 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 373 election: Election decided. Result: candidate lost.
I1014 19:26:48.594367 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:48.594421 15762 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Leader election lost for term 373. Reason: could not achieve majority
I1014 19:26:48.594715 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 373 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 373: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:48.594722 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 373 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.594867 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 373 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.595208 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 373 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 373 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.595275 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 373 election: Election decided. Result: candidate lost.
I1014 19:26:48.595450 16480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:48.595502 16480 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Leader election lost for term 373. Reason: could not achieve majority
I1014 19:26:48.603672 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 372. Current term is 373. Ops: []
I1014 19:26:48.604545 16143 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 372.246, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I1014 19:26:48.605195 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 372. Current term is 373. Ops: []
I1014 19:26:48.605149 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 372 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.605346 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.605490 16354 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Stepping down as leader of term 372
I1014 19:26:48.605530 16354 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.605700 16354 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 372.246, Last appended by leader: 246, Current term: 372, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.605875 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:48.605921 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:48.606027 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 372 FOLLOWER]: Advancing to term 373
I1014 19:26:48.686987 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 373 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:48.687088 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:48.687142 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 373 FOLLOWER]: Advancing to term 374
I1014 19:26:48.689299 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.689807 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 374 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.689963 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 374 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.690587 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 374 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.690618 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 374 candidate_status { last_received { term: 372 index: 246 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.690738 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 373 FOLLOWER]: Advancing to term 374
I1014 19:26:48.690762 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 373 FOLLOWER]: Advancing to term 374
I1014 19:26:48.690806 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.159s (vote granted)
I1014 19:26:48.690824 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.159s (vote granted)
I1014 19:26:48.692737 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 374 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 374.
I1014 19:26:48.693159 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 374 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.693218 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 374 election: Election decided. Result: candidate won.
I1014 19:26:48.693442 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:48.693471 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 374 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 374.
I1014 19:26:48.693604 16354 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 FOLLOWER]: Leader election won for term 374
I1014 19:26:48.693878 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 374 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.693909 16354 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.694262 16354 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 372.246, Last appended by leader: 246, Current term: 374, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.696084 16810 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.696254 16810 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.773918 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 374 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 372 index: 246. Preceding OpId from leader: term: 374 index: 247. (index mismatch)
I1014 19:26:48.774456 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I1014 19:26:48.775722 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.775928 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.776870 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.777076 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:48.777760 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 374 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 372 index: 246. Preceding OpId from leader: term: 374 index: 247. (index mismatch)
I1014 19:26:48.778106 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.778280 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.778491 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I1014 19:26:48.779929 16706 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.779906 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.780102 16706 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.780113 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:48.817291 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:48.818404 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:48.819851 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 370 vs 374: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.820271 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:48.820575 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 374 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:48.878403 15762 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 374 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:48.878515 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:48.878568 15762 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 374 FOLLOWER]: Advancing to term 375
I1014 19:26:48.880738 15762 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.881222 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 375 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:48.881371 15762 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 375 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.882072 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 375 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.882169 16706 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 374 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:48.882230 16706 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:48.882269 16706 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 374 FOLLOWER]: Advancing to term 375
I1014 19:26:48.882275 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 375 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.882740 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 375 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 375 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.884235 16706 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.884706 16706 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 375 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.884837 16706 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 375 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.885046 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 375 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:48.885195 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 375: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.885447 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 375 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.885430 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 375 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.885608 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 375 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.885622 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 375: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:48.885628 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 375 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 375: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:48.885684 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 375 election: Election decided. Result: candidate lost.
I1014 19:26:48.886091 15762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:48.886152 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 375 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 375 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:48.886175 15762 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Leader election lost for term 375. Reason: could not achieve majority
I1014 19:26:48.886320 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 375 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 375: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:48.886384 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 375 election: Election decided. Result: candidate lost.
I1014 19:26:48.886580 16706 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (election complete)
I1014 19:26:48.886632 16706 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Leader election lost for term 375. Reason: could not achieve majority
I1014 19:26:48.895211 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 374. Current term is 375. Ops: []
I1014 19:26:48.896283 16354 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 374.247, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I1014 19:26:48.896540 16354 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Stepping down as leader of term 374
I1014 19:26:48.896587 16354 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:48.896776 16354 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 374.247, Last appended by leader: 247, Current term: 374, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.897061 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:48.897114 16354 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:48.897254 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 374 FOLLOWER]: Advancing to term 375
I1014 19:26:48.993723 16706 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:48.993815 16706 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.201s (starting election)
I1014 19:26:48.993873 16706 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 375 FOLLOWER]: Advancing to term 376
I1014 19:26:48.995481 16706 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:48.996049 16706 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 376 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:48.996213 16706 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 376 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.996945 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 376 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:48.996955 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 376 candidate_status { last_received { term: 374 index: 247 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:48.997113 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 375 FOLLOWER]: Advancing to term 376
I1014 19:26:48.997113 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 375 FOLLOWER]: Advancing to term 376
I1014 19:26:48.997196 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (vote granted)
I1014 19:26:48.997195 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.186s (vote granted)
I1014 19:26:48.999362 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 376.
I1014 19:26:48.999362 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 376 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 376.
I1014 19:26:48.999768 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 376 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:48.999840 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 376 election: Election decided. Result: candidate won.
I1014 19:26:48.999904 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 376 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.000051 16706 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.200s (election complete)
I1014 19:26:49.000108 16706 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 FOLLOWER]: Leader election won for term 376
I1014 19:26:49.000190 16706 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:49.000380 16706 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 374.247, Last appended by leader: 247, Current term: 376, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.001418 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.001579 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.001746 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:49.003839 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:49.085861 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 376 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 374 index: 247. Preceding OpId from leader: term: 376 index: 248. (index mismatch)
I1014 19:26:49.086328 16480 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I1014 19:26:49.087535 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.087740 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.091387 16706 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.091595 16706 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.091902 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.092079 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.092218 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 374 index: 247. Preceding OpId from leader: term: 376 index: 248. (index mismatch)
I1014 19:26:49.092694 16706 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I1014 19:26:49.094483 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.094648 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.094921 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.095136 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.183147 16706 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.183382 16706 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:49.187475 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.952s	user 0.002s	sys 0.000s
I1014 19:26:49.187520 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:49.188169 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:49.188971 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 365 vs 376: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.189178 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.189333 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.189504 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.189659 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.189806 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 376 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.387774 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 376 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:49.387869 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:49.387919 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 376 FOLLOWER]: Advancing to term 377
I1014 19:26:49.389434 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.389833 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 377 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.389984 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 377 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.390622 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 377 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:49.390622 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 377 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:49.390815 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 377 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.390822 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 377 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.391160 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 377 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 377 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.391294 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 377 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 377 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.391357 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 377 election: Election decided. Result: candidate lost.
I1014 19:26:49.391548 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:26:49.391602 16354 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Leader election lost for term 377. Reason: could not achieve majority
I1014 19:26:49.406664 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 376. Current term is 377. Ops: []
I1014 19:26:49.407121 16706 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 376.248, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I1014 19:26:49.407371 16480 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 LEADER]: Stepping down as leader of term 376
I1014 19:26:49.407420 16480 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:49.407627 16480 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 376.248, Last appended by leader: 248, Current term: 376, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.407811 16480 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:49.407855 16480 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:49.407976 16480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 376 FOLLOWER]: Advancing to term 377
I1014 19:26:49.444748 17114 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:49.444830 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:49.444885 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 376 FOLLOWER]: Advancing to term 377
I1014 19:26:49.446923 17114 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.447422 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 377 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.447613 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 377 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:49.448534 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 377 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:49.448688 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (vote granted)
I1014 19:26:49.450670 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 377 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 377.
I1014 19:26:49.451433 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 377 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.451511 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 377 election: Election decided. Result: candidate won.
I1014 19:26:49.451856 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:49.451864 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 377 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:49.451948 17114 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 FOLLOWER]: Leader election won for term 377
I1014 19:26:49.452052 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 377: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:49.452270 17114 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:49.452401 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 377 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 377: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:49.452585 17114 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 376.248, Last appended by leader: 248, Current term: 377, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.453809 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 377 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.453979 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.454151 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:49.455950 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:49.515869 16354 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:49.515965 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.150s (starting election)
I1014 19:26:49.516021 16354 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 377 FOLLOWER]: Advancing to term 378
I1014 19:26:49.518208 16354 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.518771 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 378 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.518999 16354 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 378 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.519644 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 378 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:49.519641 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 378 candidate_status { last_received { term: 376 index: 248 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:49.519796 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 377 FOLLOWER]: Advancing to term 378
I1014 19:26:49.519842 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 378 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.519867 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:49.520359 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 378 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 378 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.521862 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 378 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 378.
I1014 19:26:49.523367 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 378 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.523463 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 378 election: Election decided. Result: candidate won.
I1014 19:26:49.523671 16354 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.220s (election complete)
I1014 19:26:49.523738 16354 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 FOLLOWER]: Leader election won for term 378
I1014 19:26:49.532754 16354 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:49.534126 16354 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 376.248, Last appended by leader: 248, Current term: 378, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.538566 17271 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.538767 17271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.561875 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 377. Current term is 378. Ops: []
I1014 19:26:49.562363 17114 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I1014 19:26:49.562585 17114 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 378.249 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 248
I1014 19:26:49.562671 17114 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 LEADER]: Stepping down as leader of term 377
I1014 19:26:49.562707 17114 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:49.562865 17114 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 377.249, Last appended by leader: 249, Current term: 377, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.563033 17114 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:49.563067 17114 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:49.563163 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 377 FOLLOWER]: Advancing to term 378
I1014 19:26:49.628041 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 378 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 376 index: 248. Preceding OpId from leader: term: 378 index: 249. (index mismatch)
I1014 19:26:49.628713 17271 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I1014 19:26:49.629853 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.630071 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.631733 17271 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.631904 17271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.632964 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 377 index: 249. Preceding OpId from leader: term: 378 index: 249. (term mismatch)
I1014 19:26:49.633020 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 248
I1014 19:26:49.633040 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 377.249
I1014 19:26:49.633077 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:49.633131 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.633299 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.634116 17271 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I1014 19:26:49.641497 15762 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.641688 15762 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.641806 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.641957 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.715034 17271 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 378 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.715284 17271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.732091 16480 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 378 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:49.732177 16480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:26:49.732234 16480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 378 FOLLOWER]: Advancing to term 379
I1014 19:26:49.734413 16480 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.734923 16480 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 379 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.735536 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 379 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:49.735680 16480 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 379 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:49.735710 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 379 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.736274 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 379 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:49.736464 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 379 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.736636 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 379 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 379 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.737439 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 379 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 379 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:49.737534 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 379 election: Election decided. Result: candidate lost.
I1014 19:26:49.737749 16480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:49.737800 16480 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Leader election lost for term 379. Reason: could not achieve majority
I1014 19:26:49.741497 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 378. Current term is 379. Ops: []
I1014 19:26:49.741910 17280 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 378.249, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I1014 19:26:49.742120 17271 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Stepping down as leader of term 378
I1014 19:26:49.742166 17271 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:49.742319 17271 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 378.249, Last appended by leader: 249, Current term: 378, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.742445 17271 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:49.742480 17271 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:49.742596 17271 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 378 FOLLOWER]: Advancing to term 379
W1014 19:26:49.785581 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.378s	user 0.002s	sys 0.000s
I1014 19:26:49.785639 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:49.786518 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:49.787510 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 372 vs 379: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.787714 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.787845 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.787978 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.788136 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.788343 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.788519 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:49.788694 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 379 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.816227 17114 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:49.816313 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:49.816367 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 378 FOLLOWER]: Advancing to term 379
I1014 19:26:49.818573 17114 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 379 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.819046 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 379 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.819383 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 379 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:49.819365 17271 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 379 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:49.819435 17271 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:49.819499 17271 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 379 FOLLOWER]: Advancing to term 380
I1014 19:26:49.820022 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 379 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:49.821589 17271 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.821910 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 379. Current term is 380.
I1014 19:26:49.822046 17271 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 380 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.822190 17271 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 380 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:26:49.822383 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 379 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 379. Current term is 380.
I1014 19:26:49.822419 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 379 election: Cancelling election due to peer responding with higher term
I1014 19:26:49.822669 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:49.822726 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 379 FOLLOWER]: Advancing to term 380
I1014 19:26:49.822705 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 379 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:49.822902 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 379: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:49.823050 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 380 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:49.823166 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Advancing to term 380
I1014 19:26:49.823222 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.164s (vote granted)
I1014 19:26:49.824753 17114 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 380 FOLLOWER]: Leader election lost for term 379. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 379. Current term is 380.
I1014 19:26:49.825155 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 380 candidate_status { last_received { term: 378 index: 249 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:49.825261 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 380.
I1014 19:26:49.825295 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.209s (vote granted)
I1014 19:26:49.825781 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 380 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:49.825850 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 380 election: Election decided. Result: candidate won.
I1014 19:26:49.826243 17271 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:26:49.826303 17271 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Leader election won for term 380
I1014 19:26:49.826478 17271 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:49.826779 17271 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 378.249, Last appended by leader: 249, Current term: 380, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:49.827314 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 380 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 380.
I1014 19:26:49.827630 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 379 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 379 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 379: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:49.827884 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.828071 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.828068 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 380 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:49.828258 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:49.830070 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:49.934721 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 380 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 378 index: 249. Preceding OpId from leader: term: 380 index: 250. (index mismatch)
I1014 19:26:49.935176 17271 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I1014 19:26:49.936614 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.936827 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.937723 17271 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.937932 17271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:49.938416 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 378 index: 249. Preceding OpId from leader: term: 380 index: 250. (index mismatch)
I1014 19:26:49.939002 17271 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I1014 19:26:49.938997 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.939160 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.940697 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.940907 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:49.941043 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:49.941184 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.020720 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 380 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.020936 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.039240 17114 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 380 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:50.039317 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:50.039366 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 380 FOLLOWER]: Advancing to term 381
I1014 19:26:50.041013 17114 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.041509 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 381 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:50.041712 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 381 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.042311 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 381 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:50.042368 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 381 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.042497 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 381 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.042578 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 381 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.042809 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 381 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 381 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.042954 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 381 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 381 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.043009 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 381 election: Election decided. Result: candidate lost.
I1014 19:26:50.043165 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:50.043217 17114 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Leader election lost for term 381. Reason: could not achieve majority
I1014 19:26:50.058837 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 380. Current term is 381. Ops: []
I1014 19:26:50.059326 16354 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 380.250, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I1014 19:26:50.059608 17280 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 LEADER]: Stepping down as leader of term 380
I1014 19:26:50.059667 17280 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:50.059837 17280 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 380.250, Last appended by leader: 250, Current term: 380, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.059985 17280 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:50.060026 17280 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:50.060155 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 380 FOLLOWER]: Advancing to term 381
I1014 19:26:50.122835 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:50.122951 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:50.123020 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 380 FOLLOWER]: Advancing to term 381
I1014 19:26:50.124701 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.125075 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 381 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.125216 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 381 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.126077 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 381 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:50.126245 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 381: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:50.126236 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 381 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.126399 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (vote granted)
I1014 19:26:50.126648 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 381 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 381: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:50.128499 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 381 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 381.
I1014 19:26:50.128902 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 381 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.128973 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 381 election: Election decided. Result: candidate won.
I1014 19:26:50.129216 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:26:50.129298 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 FOLLOWER]: Leader election won for term 381
I1014 19:26:50.129402 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:50.129653 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 380.250, Last appended by leader: 250, Current term: 381, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.130522 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 381 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.130712 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.158538 17114 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:50.158625 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.179s (starting election)
I1014 19:26:50.158690 17114 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 381 FOLLOWER]: Advancing to term 382
I1014 19:26:50.160710 17114 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.161141 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 382 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:50.161269 17114 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 382 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.165868 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 382 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.166054 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 381 FOLLOWER]: Advancing to term 382
I1014 19:26:50.166128 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (vote granted)
I1014 19:26:50.166242 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 382 candidate_status { last_received { term: 380 index: 250 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:50.166419 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 382 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.167065 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 382 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 382 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.168169 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 382 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 382.
I1014 19:26:50.169062 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 382 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.169111 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 382 election: Election decided. Result: candidate won.
I1014 19:26:50.169411 17114 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:50.169495 17114 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 FOLLOWER]: Leader election won for term 382
I1014 19:26:50.180229 17114 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:50.180591 17114 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 380.250, Last appended by leader: 250, Current term: 382, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.182535 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.182703 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.231794 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 381. Current term is 382. Ops: []
I1014 19:26:50.232431 16480 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I1014 19:26:50.232717 16480 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 382.251 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 250
I1014 19:26:50.232815 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 LEADER]: Stepping down as leader of term 381
I1014 19:26:50.232897 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:50.233108 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 381.251, Last appended by leader: 251, Current term: 381, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.233311 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:50.233359 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:50.233505 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 381 FOLLOWER]: Advancing to term 382
I1014 19:26:50.261399 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 381 index: 251. Preceding OpId from leader: term: 382 index: 251. (term mismatch)
I1014 19:26:50.261474 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 250
I1014 19:26:50.261503 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 381.251
I1014 19:26:50.261565 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:50.262063 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I1014 19:26:50.263200 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.263473 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.265126 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.265363 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.266032 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 382 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 380 index: 250. Preceding OpId from leader: term: 382 index: 251. (index mismatch)
I1014 19:26:50.266731 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.267021 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.267405 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I1014 19:26:50.268707 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.268898 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.268995 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.269157 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.371299 17280 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 382 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:50.371382 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:50.371428 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 382 FOLLOWER]: Advancing to term 383
I1014 19:26:50.373615 17280 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.374039 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 383 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.374217 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 383 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:50.374826 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 382 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.374919 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 383 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:50.375087 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.375111 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 383 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.375201 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 382. Current term is 383. Ops: []
I1014 19:26:50.375334 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 383 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:50.375485 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 383 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.375718 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 383 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 383 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.375718 17114 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 382.251, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I1014 19:26:50.375891 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 383 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 383 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.375960 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 383 election: Election decided. Result: candidate lost.
I1014 19:26:50.376055 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Stepping down as leader of term 382
I1014 19:26:50.376091 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:50.376370 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:50.376251 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 382.251, Last appended by leader: 251, Current term: 382, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.376425 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:50.376497 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:50.376485 17280 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Leader election lost for term 383. Reason: could not achieve majority
I1014 19:26:50.376591 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 382 FOLLOWER]: Advancing to term 383
I1014 19:26:50.446858 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:50.446967 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:26:50.447046 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 382 FOLLOWER]: Advancing to term 383
I1014 19:26:50.448789 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.449208 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 383 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.449388 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 383 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.450820 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 383 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:50.450961 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:50.451061 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 383 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.451251 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 383: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:50.451623 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 383 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 383: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:50.452723 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 383 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 383.
I1014 19:26:50.453022 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 383 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.453084 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 383 election: Election decided. Result: candidate won.
I1014 19:26:50.453325 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:50.453397 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 FOLLOWER]: Leader election won for term 383
I1014 19:26:50.453521 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:50.453703 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 382.251, Last appended by leader: 251, Current term: 383, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.454819 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 383 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.455003 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:50.496809 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.041s	user 0.002s	sys 0.000s
I1014 19:26:50.496850 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:50.497709 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:50.498751 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 377 vs 383: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 383 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.498966 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 383 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.499132 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 383 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.499294 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 383 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.515794 17280 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:50.515879 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.166s (starting election)
I1014 19:26:50.515938 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 383 FOLLOWER]: Advancing to term 384
I1014 19:26:50.518126 17280 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.518688 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 384 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.518854 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 384 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:50.519719 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 384 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:50.519914 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 383 FOLLOWER]: Advancing to term 384
I1014 19:26:50.519999 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:50.522262 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 384 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 384.
I1014 19:26:50.522577 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 384 candidate_status { last_received { term: 382 index: 251 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:50.522766 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 384 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.523631 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 384 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 384 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.523761 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 384 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.523819 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 384 election: Election decided. Result: candidate won.
I1014 19:26:50.525084 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:26:50.525154 17280 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 FOLLOWER]: Leader election won for term 384
I1014 19:26:50.525257 17280 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:50.525651 17280 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 382.251, Last appended by leader: 251, Current term: 384, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.529961 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.530120 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.542204 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 384 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 383. Current term is 384. Ops: []
I1014 19:26:50.542989 16480 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I1014 19:26:50.543303 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 LEADER]: Stepping down as leader of term 383
I1014 19:26:50.543364 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:50.543643 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 383.252, Last appended by leader: 252, Current term: 383, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.543853 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:50.543928 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:50.544054 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 383 FOLLOWER]: Advancing to term 384
I1014 19:26:50.613536 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 383 index: 252. Preceding OpId from leader: term: 384 index: 252. (term mismatch)
I1014 19:26:50.613605 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 251
I1014 19:26:50.613663 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 383.252
I1014 19:26:50.613719 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:50.614264 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I1014 19:26:50.615722 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.615880 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.616925 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.617132 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.618237 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 384 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 382 index: 251. Preceding OpId from leader: term: 384 index: 252. (index mismatch)
I1014 19:26:50.618674 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.618901 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.618906 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s
I1014 19:26:50.620311 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.620611 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.620549 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.620738 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.720402 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 384 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:50.720485 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:50.720554 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 384 FOLLOWER]: Advancing to term 385
I1014 19:26:50.721662 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.721886 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.722070 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.722434 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 384. Current term is 385. Ops: []
I1014 19:26:50.722662 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 385 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:50.722800 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 385 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.723019 17280 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 384.252, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I1014 19:26:50.723284 17280 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 LEADER]: Stepping down as leader of term 384
I1014 19:26:50.723323 17280 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:50.723325 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 385 candidate_status { last_received { term: 384 index: 252 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:50.723487 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 385 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.723392 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 385 candidate_status { last_received { term: 384 index: 252 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.723525 17280 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 384.252, Last appended by leader: 252, Current term: 384, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.723695 17280 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:50.723737 17280 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:50.723816 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 385 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 385 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.723871 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 384 FOLLOWER]: Advancing to term 385
I1014 19:26:50.725695 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (vote granted)
I1014 19:26:50.727002 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 385.
I1014 19:26:50.727459 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 385 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.727515 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 385 election: Election decided. Result: candidate won.
I1014 19:26:50.727718 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:26:50.727799 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 FOLLOWER]: Leader election won for term 385
I1014 19:26:50.727881 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:50.728061 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 384.252, Last appended by leader: 252, Current term: 385, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.728966 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.729142 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.729301 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:50.731199 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:50.794318 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.790s	user 0.000s	sys 0.001s
I1014 19:26:50.794414 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:50.795205 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:50.796205 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 376 vs 384: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.796603 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.796903 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.797164 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.797416 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 384 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.802482 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:50.802562 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:50.802635 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 384 FOLLOWER]: Advancing to term 385
I1014 19:26:50.804620 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:50.804987 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 385 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:50.805167 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 385 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:50.805707 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 385 candidate_status { last_received { term: 384 index: 252 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:50.805896 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 385 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.805843 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 385 candidate_status { last_received { term: 384 index: 252 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:50.806066 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 385: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:50.806465 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 385 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 385: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:50.806617 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 385 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 385 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:50.806692 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 385 election: Election decided. Result: candidate lost.
I1014 19:26:50.806898 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:26:50.806973 17409 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Leader election lost for term 385. Reason: could not achieve majority
I1014 19:26:50.832830 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 384 index: 252. Preceding OpId from leader: term: 385 index: 253. (index mismatch)
I1014 19:26:50.833312 17114 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I1014 19:26:50.834556 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.834750 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.835547 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.835680 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:50.836153 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 384 index: 252. Preceding OpId from leader: term: 385 index: 253. (index mismatch)
I1014 19:26:50.836352 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.836522 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.836622 17114 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I1014 19:26:50.838007 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.837973 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.838219 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.838275 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:50.920375 17114 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:50.920626 17114 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:50.937116 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.107s	user 0.001s	sys 0.000s
I1014 19:26:50.937170 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:50.938138 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:50.939538 12420 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 380 vs 385: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.939939 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.940215 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.940546 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.940837 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:50.941095 12420 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 385 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.018748 17280 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:51.018836 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:51.018891 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 385 FOLLOWER]: Advancing to term 386
I1014 19:26:51.021216 17280 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.021755 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 386 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.022120 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 386 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.022738 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 386 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.022872 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 386 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.022933 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 386 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.023041 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 386 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.023429 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 386 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 386 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.023581 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 386 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 386 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.023658 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 386 election: Election decided. Result: candidate lost.
I1014 19:26:51.023949 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:51.024011 17280 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Leader election lost for term 386. Reason: could not achieve majority
I1014 19:26:51.027295 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 385. Current term is 386. Ops: []
I1014 19:26:51.027775 17540 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 385.253, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I1014 19:26:51.028084 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Stepping down as leader of term 385
I1014 19:26:51.028125 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:51.028281 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 385.253, Last appended by leader: 253, Current term: 385, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.028412 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:51.028455 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:51.028590 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 385 FOLLOWER]: Advancing to term 386
I1014 19:26:51.102474 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:51.102571 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:51.102648 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 385 FOLLOWER]: Advancing to term 386
I1014 19:26:51.104380 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.104820 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 386 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.105023 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 386 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.105512 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 386 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.105662 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:51.105707 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 386 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.105895 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 386: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:51.106343 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 386 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 386: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:51.107285 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 386 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 386.
I1014 19:26:51.107688 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 386 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.107753 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 386 election: Election decided. Result: candidate won.
I1014 19:26:51.108137 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:51.108209 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 FOLLOWER]: Leader election won for term 386
I1014 19:26:51.108330 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.108538 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 385.253, Last appended by leader: 253, Current term: 386, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.109479 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 386 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.109639 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.109849 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:51.111301 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:51.127701 17280 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:51.127795 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.176s (starting election)
I1014 19:26:51.127879 17280 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 386 FOLLOWER]: Advancing to term 387
I1014 19:26:51.129695 17280 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.130095 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 387 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.130250 17280 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 387 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.130851 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 387 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.131021 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 386 FOLLOWER]: Advancing to term 387
I1014 19:26:51.130959 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 387 candidate_status { last_received { term: 385 index: 253 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.131086 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (vote granted)
I1014 19:26:51.131170 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 387 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.131595 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 387 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 387 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.132728 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 387.
I1014 19:26:51.133060 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 387 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.133121 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 387 election: Election decided. Result: candidate won.
I1014 19:26:51.133329 17280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:51.133388 17280 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 FOLLOWER]: Leader election won for term 387
I1014 19:26:51.133497 17280 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:51.133677 17280 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 385.253, Last appended by leader: 253, Current term: 387, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.134528 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.134768 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.134932 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:51.136317 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:51.212589 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 386. Current term is 387. Ops: []
I1014 19:26:51.213327 16480 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I1014 19:26:51.213655 16480 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 387.254 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 253
I1014 19:26:51.213798 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 LEADER]: Stepping down as leader of term 386
I1014 19:26:51.213856 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.214090 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 386.254, Last appended by leader: 254, Current term: 386, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.214295 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:51.214357 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:51.214505 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 386 FOLLOWER]: Advancing to term 387
I1014 19:26:51.220202 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 387 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 386 index: 254. Preceding OpId from leader: term: 387 index: 254. (term mismatch)
I1014 19:26:51.220260 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 253
I1014 19:26:51.220293 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 386.254
I1014 19:26:51.220342 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 387 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:51.221514 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 385 index: 253. Preceding OpId from leader: term: 387 index: 254. (index mismatch)
I1014 19:26:51.222056 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I1014 19:26:51.222882 16354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I1014 19:26:51.223522 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.223709 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.224530 17280 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.224712 17280 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.224661 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.224920 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.225966 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.226147 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.226747 16354 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.226917 16354 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.227977 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 387 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.228159 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.328269 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 387 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:51.328373 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:26:51.328444 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 387 FOLLOWER]: Advancing to term 388
I1014 19:26:51.330814 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.331348 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 388 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.331878 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 387. Current term is 388. Ops: []
I1014 19:26:51.332041 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 388 candidate_status { last_received { term: 387 index: 254 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.332229 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 388 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.332638 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 388 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 388 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.333034 17931 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 387.254, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I1014 19:26:51.333353 17931 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 LEADER]: Stepping down as leader of term 387
I1014 19:26:51.333436 17931 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:51.333729 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 388 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.333652 17931 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 387.254, Last appended by leader: 254, Current term: 387, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.333843 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:51.333884 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:51.334013 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 387 FOLLOWER]: Advancing to term 388
I1014 19:26:51.334435 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 388 candidate_status { last_received { term: 387 index: 254 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.336892 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:26:51.338924 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 388.
I1014 19:26:51.341321 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 388 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.341408 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 388 election: Election decided. Result: candidate won.
I1014 19:26:51.342403 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:26:51.342504 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 FOLLOWER]: Leader election won for term 388
I1014 19:26:51.342650 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.342875 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 387.254, Last appended by leader: 254, Current term: 388, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.343907 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.344151 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.407606 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:51.407687 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:51.407739 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 387 FOLLOWER]: Advancing to term 388
I1014 19:26:51.409355 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 388 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.409767 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 388 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.409924 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 388 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.410466 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 388 candidate_status { last_received { term: 387 index: 254 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.410478 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 388 candidate_status { last_received { term: 387 index: 254 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.410657 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 388: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:51.410697 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 388 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.410998 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 388 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 388: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:51.411095 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 388 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 388 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.411192 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 388 election: Election decided. Result: candidate lost.
I1014 19:26:51.411383 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:51.411464 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 388 FOLLOWER]: Leader election lost for term 388. Reason: could not achieve majority
I1014 19:26:51.451148 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 387 index: 254. Preceding OpId from leader: term: 388 index: 255. (index mismatch)
I1014 19:26:51.451706 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 388 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 387 index: 254. Preceding OpId from leader: term: 388 index: 255. (index mismatch)
I1014 19:26:51.451792 16480 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I1014 19:26:51.452201 16480 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s
I1014 19:26:51.453562 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.453758 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.453722 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.453929 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.454947 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.455145 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.455345 16480 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.455400 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.455548 16480 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.455585 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.456137 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 388 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.456336 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.555948 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 388 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:51.556030 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:51.556082 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 388 FOLLOWER]: Advancing to term 389
I1014 19:26:51.556447 17931 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:51.556514 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:51.556555 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 388 FOLLOWER]: Advancing to term 389
I1014 19:26:51.558213 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 389 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.558714 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 389 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.558692 17931 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.558890 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 389 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.559129 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 389 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.559263 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 389 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.559509 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 389 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.559751 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 389 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.559737 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 389 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.559782 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 389 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.559934 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 389: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:51.559948 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 389 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 389: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:51.560195 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 389 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 389 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.560194 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 389 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.560281 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 389 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 389: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:51.560305 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 389 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 389 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 389: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:51.560335 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 389 election: Election decided. Result: candidate lost.
I1014 19:26:51.560356 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 389 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.560586 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:51.560637 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 389 FOLLOWER]: Leader election lost for term 389. Reason: could not achieve majority
I1014 19:26:51.560701 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 389 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 389 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.560775 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 389 election: Election decided. Result: candidate lost.
I1014 19:26:51.560950 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:51.561003 17931 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Leader election lost for term 389. Reason: could not achieve majority
I1014 19:26:51.562093 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 388. Current term is 389. Ops: []
I1014 19:26:51.562567 17938 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 388.255, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I1014 19:26:51.562799 17938 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Stepping down as leader of term 388
I1014 19:26:51.562856 17938 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.563055 17938 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 388.255, Last appended by leader: 255, Current term: 388, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.563275 17938 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:51.563334 17938 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:51.563468 17938 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 388 FOLLOWER]: Advancing to term 389
I1014 19:26:51.657099 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 389 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:51.657214 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:26:51.657294 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 389 FOLLOWER]: Advancing to term 390
I1014 19:26:51.659613 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.660166 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 390 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.660583 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 390 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.660961 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 390 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.661124 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 389 FOLLOWER]: Advancing to term 390
I1014 19:26:51.661186 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (vote granted)
I1014 19:26:51.663367 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 390 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 390.
I1014 19:26:51.663923 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 390 candidate_status { last_received { term: 388 index: 255 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.664081 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 389 FOLLOWER]: Advancing to term 390
I1014 19:26:51.664160 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.219s (vote granted)
I1014 19:26:51.664247 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 390 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.664316 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 390 election: Election decided. Result: candidate won.
I1014 19:26:51.664571 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:51.664643 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 FOLLOWER]: Leader election won for term 390
I1014 19:26:51.664786 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.665000 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 388.255, Last appended by leader: 255, Current term: 390, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.666059 17938 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.666301 17938 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.666376 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 390.
I1014 19:26:51.666734 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 390 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.777287 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 390 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 388 index: 255. Preceding OpId from leader: term: 390 index: 256. (index mismatch)
I1014 19:26:51.777901 17938 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I1014 19:26:51.778148 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 388 index: 255. Preceding OpId from leader: term: 390 index: 256. (index mismatch)
I1014 19:26:51.778722 17938 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I1014 19:26:51.779237 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.779450 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.779779 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.780092 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.780634 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.780810 17938 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.781000 17938 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.780939 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:51.781545 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.781728 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.781850 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 390 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.782021 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:51.882427 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 390 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:51.882511 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:51.882557 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 390 FOLLOWER]: Advancing to term 391
I1014 19:26:51.884186 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.884573 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 391 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.884686 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 391 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:51.885143 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 391 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.885164 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 391 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:51.885349 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 391 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.885377 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 391 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.885686 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 391 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 391 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.885758 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 391 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 391 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:51.885824 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 391 election: Election decided. Result: candidate lost.
I1014 19:26:51.885987 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:51.886040 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Leader election lost for term 391. Reason: could not achieve majority
I1014 19:26:51.888792 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 390. Current term is 391. Ops: []
I1014 19:26:51.889206 17938 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 390.256, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I1014 19:26:51.889456 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 LEADER]: Stepping down as leader of term 390
I1014 19:26:51.889511 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:51.889700 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 390.256, Last appended by leader: 256, Current term: 390, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.889840 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:51.889899 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:51.890009 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 390 FOLLOWER]: Advancing to term 391
I1014 19:26:51.980175 17931 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:51.980303 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:26:51.980365 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 390 FOLLOWER]: Advancing to term 391
I1014 19:26:51.982733 17931 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.983271 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 391 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:51.983449 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 391 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.984163 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 391 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:51.984175 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 391 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:51.984367 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 391: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:51.984375 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (vote granted)
I1014 19:26:51.984897 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 391 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 391: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:51.986609 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 391 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 391.
I1014 19:26:51.986968 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 391 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:51.987023 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 391 election: Election decided. Result: candidate won.
I1014 19:26:51.987282 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:51.987360 17931 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 FOLLOWER]: Leader election won for term 391
I1014 19:26:51.992007 17931 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:51.992565 17931 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 390.256, Last appended by leader: 256, Current term: 391, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:51.994899 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 391 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:51.995347 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.006646 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:52.006767 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.205s (starting election)
I1014 19:26:52.006830 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 391 FOLLOWER]: Advancing to term 392
I1014 19:26:52.009207 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.009802 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 392 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.009984 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 392 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.010627 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 392 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.010639 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 392 candidate_status { last_received { term: 390 index: 256 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.010802 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 391 FOLLOWER]: Advancing to term 392
I1014 19:26:52.010848 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 392 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.010890 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:52.011380 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 392 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 392 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.013160 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 392 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 392.
I1014 19:26:52.013933 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 392 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.013993 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 392 election: Election decided. Result: candidate won.
I1014 19:26:52.014343 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:26:52.014406 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 FOLLOWER]: Leader election won for term 392
I1014 19:26:52.022511 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.022825 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 390.256, Last appended by leader: 256, Current term: 392, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.024238 18199 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.024703 18199 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.085886 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 392 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 391. Current term is 392. Ops: []
I1014 19:26:52.086511 18198 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I1014 19:26:52.086748 17931 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 LEADER]: Stepping down as leader of term 391
I1014 19:26:52.086802 17931 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:52.086998 17931 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 391.257, Last appended by leader: 257, Current term: 391, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.087285 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.087329 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:52.087481 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 391 FOLLOWER]: Advancing to term 392
I1014 19:26:52.112083 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 391 index: 257. Preceding OpId from leader: term: 392 index: 257. (term mismatch)
I1014 19:26:52.112133 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 256
I1014 19:26:52.112152 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 391.257
I1014 19:26:52.112190 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:52.112757 18199 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I1014 19:26:52.113824 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.114054 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.115074 18199 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.115247 18199 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.115372 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.115571 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.115998 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 392 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 390 index: 256. Preceding OpId from leader: term: 392 index: 257. (index mismatch)
I1014 19:26:52.116492 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s
I1014 19:26:52.117813 17938 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.117964 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.118131 17938 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.118161 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.218255 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 392 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:52.218360 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:52.218463 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 392 FOLLOWER]: Advancing to term 393
I1014 19:26:52.220731 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.221240 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 393 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.221629 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 393 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.222285 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 393 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.222555 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 393 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.222851 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 393 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.222931 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 393 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 393 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.223022 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 393 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.223350 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 393 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 393 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.223438 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 393 election: Election decided. Result: candidate lost.
I1014 19:26:52.223656 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:26:52.223734 17409 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Leader election lost for term 393. Reason: could not achieve majority
I1014 19:26:52.241529 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 392 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.241744 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.242070 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 392. Current term is 393. Ops: []
I1014 19:26:52.242578 18285 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 392.257, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I1014 19:26:52.242921 18285 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 LEADER]: Stepping down as leader of term 392
I1014 19:26:52.242961 18285 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.243211 18285 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 392.257, Last appended by leader: 257, Current term: 392, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.243393 18285 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:52.243428 18285 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.243562 18285 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 392 FOLLOWER]: Advancing to term 393
W1014 19:26:52.273123 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.162s	user 0.002s	sys 0.000s
I1014 19:26:52.273178 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:52.274076 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:52.275142 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 386 vs 393: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.275399 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.275645 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.275872 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.277178 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.277544 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.277822 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 393 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.309042 17931 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:52.309146 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:26:52.309193 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 392 FOLLOWER]: Advancing to term 393
I1014 19:26:52.311450 17931 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.311969 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 393 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.312142 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 393 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.313021 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 393 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.313035 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 393 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.313179 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:52.313246 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 393: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:52.313707 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 393 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 393: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:52.315166 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 393 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 393.
I1014 19:26:52.315516 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 393 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.315582 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 393 election: Election decided. Result: candidate won.
I1014 19:26:52.315802 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (election complete)
I1014 19:26:52.315896 17931 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 FOLLOWER]: Leader election won for term 393
I1014 19:26:52.316031 17931 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:52.316252 17931 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 392.257, Last appended by leader: 257, Current term: 393, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.317642 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 393 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.317824 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.339511 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:52.339618 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.168s (starting election)
I1014 19:26:52.339701 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 393 FOLLOWER]: Advancing to term 394
I1014 19:26:52.342028 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.345158 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 394 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.345386 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 394 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.345885 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 394 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.346055 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 393 FOLLOWER]: Advancing to term 394
I1014 19:26:52.346127 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:26:52.348381 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 394.
I1014 19:26:52.348763 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 394 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.348747 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 394 candidate_status { last_received { term: 392 index: 257 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.348845 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 394 election: Election decided. Result: candidate won.
I1014 19:26:52.348958 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 394 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.349078 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:52.349158 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 FOLLOWER]: Leader election won for term 394
I1014 19:26:52.349258 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:52.349472 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 392.257, Last appended by leader: 257, Current term: 394, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.350569 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.350751 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.351370 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:52.353636 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:52.350951 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 394 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 394 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.397622 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 393. Current term is 394. Ops: []
I1014 19:26:52.398170 18198 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I1014 19:26:52.398459 18198 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 LEADER]: Stepping down as leader of term 393
I1014 19:26:52.398499 18198 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:52.398697 18198 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 393.258, Last appended by leader: 258, Current term: 393, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.398947 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.398996 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:52.399109 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 393 FOLLOWER]: Advancing to term 394
W1014 19:26:52.412686 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.681s	user 0.001s	sys 0.000s
I1014 19:26:52.412736 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:52.413460 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:52.414263 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 385 vs 394: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.414502 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.414671 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.414842 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.415011 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:52.415190 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 394 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.448230 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 394 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 393 index: 258. Preceding OpId from leader: term: 394 index: 258. (term mismatch)
I1014 19:26:52.448293 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 257
I1014 19:26:52.448314 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 393.258
I1014 19:26:52.448360 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 394 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:52.448957 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I1014 19:26:52.450227 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.450541 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.451686 17938 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.451879 17938 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.452142 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 392 index: 257. Preceding OpId from leader: term: 394 index: 258. (index mismatch)
I1014 19:26:52.452649 17938 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I1014 19:26:52.452424 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.453016 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.453886 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.453907 18285 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.454087 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.454119 18285 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.544008 17938 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 394 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.544278 17938 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.552968 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 394 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:52.553050 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:52.553105 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 394 FOLLOWER]: Advancing to term 395
I1014 19:26:52.554704 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.555135 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 395 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.555303 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 395 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.555866 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 395 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.555874 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 395 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.556025 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 395 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.556025 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 395 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.556397 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 395 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 395 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.556591 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 395 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 395 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.556658 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 395 election: Election decided. Result: candidate lost.
I1014 19:26:52.556828 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:52.556891 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Leader election lost for term 395. Reason: could not achieve majority
I1014 19:26:52.568564 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 394. Current term is 395. Ops: []
I1014 19:26:52.569030 18340 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 394.258, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I1014 19:26:52.569628 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 LEADER]: Stepping down as leader of term 394
I1014 19:26:52.569672 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:52.569854 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 394.258, Last appended by leader: 258, Current term: 394, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.569983 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:52.570041 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:52.570147 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 394 FOLLOWER]: Advancing to term 395
I1014 19:26:52.645253 18285 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:52.645340 18285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:26:52.645396 18285 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 394 FOLLOWER]: Advancing to term 395
I1014 19:26:52.647703 18285 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.648219 18285 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 395 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.648550 18285 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 395 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.649121 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 395 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.649286 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:52.650758 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 395 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 395.
I1014 19:26:52.651868 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 395 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.652089 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 395: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:52.652498 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 395 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 395: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:52.652966 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 395 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.653021 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 395 election: Election decided. Result: candidate won.
I1014 19:26:52.654207 18285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:52.654299 18285 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 FOLLOWER]: Leader election won for term 395
I1014 19:26:52.654422 18285 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.654659 18285 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 394.258, Last appended by leader: 258, Current term: 395, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.655807 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 395 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.656008 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.656177 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:52.657763 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:52.657824 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.181s (starting election)
I1014 19:26:52.657869 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 395 FOLLOWER]: Advancing to term 396
I1014 19:26:52.658169 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:52.659467 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.659919 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 396 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.660076 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 396 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.660773 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 396 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.660933 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 396 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.661159 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 396 candidate_status { last_received { term: 394 index: 258 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.661309 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 395 FOLLOWER]: Advancing to term 396
I1014 19:26:52.661375 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:52.663405 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 396 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 396.
I1014 19:26:52.663960 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 396 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.664017 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 396 election: Election decided. Result: candidate won.
I1014 19:26:52.664286 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.174s (election complete)
I1014 19:26:52.664346 18198 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 FOLLOWER]: Leader election won for term 396
I1014 19:26:52.664312 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 396 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 396 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.664470 18198 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:52.664685 18198 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 394.258, Last appended by leader: 258, Current term: 396, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.666003 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.666183 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:52.692893 12420 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.557s	user 0.002s	sys 0.000s
I1014 19:26:52.692945 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:52.693886 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:52.696597 28640 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 LEADER]: Stepping down as leader of term 395
I1014 19:26:52.696647 28640 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.696643 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 396 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 394 index: 258. Preceding OpId from leader: term: 396 index: 260. (index mismatch)
I1014 19:26:52.696878 28640 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 395.259, Last appended by leader: 259, Current term: 395, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.697101 28640 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:52.697136 28640 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.697211 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 395 FOLLOWER]: Advancing to term 396
I1014 19:26:52.697177 18198 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I1014 19:26:52.699620 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 395 index: 259. Preceding OpId from leader: term: 396 index: 260. (index mismatch)
I1014 19:26:52.699916 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.700093 17278 mvcc.cc:203] Tried to move safe_time back from 6306080410398486528 to 6306080410278633472. Current Snapshot: MvccSnapshot[committed={T|T < 6306080410398486528}]
I1014 19:26:52.700114 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.700230 17931 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I1014 19:26:52.700341 17931 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 395.259 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 258
I1014 19:26:52.704320 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 258
I1014 19:26:52.704430 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 395.259
I1014 19:26:52.704550 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:52.705085 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.705647 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.705868 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:52.708655 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:52.712311 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.712566 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.712796 18285 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.713027 18285 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.716775 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.716976 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.796788 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 396 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.797024 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.800283 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 396 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:52.800359 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:52.800415 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 396 FOLLOWER]: Advancing to term 397
I1014 19:26:52.802268 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.802675 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 397 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.802812 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 397 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.803280 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 397 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.803377 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 397 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.803503 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 397 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.803581 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 397 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.804008 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 397 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 397 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.804106 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 397 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 397 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.804183 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 397 election: Election decided. Result: candidate lost.
I1014 19:26:52.804359 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:52.804411 17409 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Leader election lost for term 397. Reason: could not achieve majority
I1014 19:26:52.823068 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 396. Current term is 397. Ops: []
I1014 19:26:52.823644 18463 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 396.260, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I1014 19:26:52.823925 17931 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 LEADER]: Stepping down as leader of term 396
I1014 19:26:52.823966 17931 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:52.824141 17931 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 396.260, Last appended by leader: 260, Current term: 396, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.824262 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.824321 17931 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:52.824440 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 396 FOLLOWER]: Advancing to term 397
I1014 19:26:52.897714 18285 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:52.897851 18285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:26:52.897927 18285 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 396 FOLLOWER]: Advancing to term 397
I1014 19:26:52.899420 18285 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.899868 18285 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 397 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:52.900104 18285 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 397 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.900977 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 397 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.901139 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:52.901192 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 397 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:52.901356 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 397: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:52.901818 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 397 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 397: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:52.903009 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 397 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 397.
I1014 19:26:52.903380 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 397 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.908990 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 397 election: Election decided. Result: candidate won.
I1014 19:26:52.909265 18285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:52.909339 18285 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 FOLLOWER]: Leader election won for term 397
I1014 19:26:52.909423 18285 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.909689 18285 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 396.260, Last appended by leader: 260, Current term: 397, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.910807 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 397 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.910945 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.950680 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:52.950769 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.198s (starting election)
I1014 19:26:52.950824 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 397 FOLLOWER]: Advancing to term 398
I1014 19:26:52.952323 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.952762 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 398 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:52.952934 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 398 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.953709 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 398 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:52.953893 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 398 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.954267 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 398 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 398 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:52.954257 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 398 candidate_status { last_received { term: 396 index: 260 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:52.954411 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 397 FOLLOWER]: Advancing to term 398
I1014 19:26:52.954512 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (vote granted)
I1014 19:26:52.956099 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 398 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 398.
I1014 19:26:52.956372 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 398 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:52.956408 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 398 election: Election decided. Result: candidate won.
I1014 19:26:52.956571 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.181s (election complete)
I1014 19:26:52.956629 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 FOLLOWER]: Leader election won for term 398
I1014 19:26:52.956702 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:52.956858 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 396.260, Last appended by leader: 260, Current term: 398, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.957706 18340 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.957866 18340 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.979566 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:52.980466 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:52.982833 28640 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 LEADER]: Stepping down as leader of term 397
I1014 19:26:52.982885 28640 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:52.982969 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 398 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 396 index: 260. Preceding OpId from leader: term: 398 index: 262. (index mismatch)
I1014 19:26:52.983078 28640 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 397.261, Last appended by leader: 261, Current term: 397, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:52.983276 28640 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:52.983317 28640 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:52.983397 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 397 FOLLOWER]: Advancing to term 398
I1014 19:26:52.983578 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I1014 19:26:52.985698 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 398 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 397 index: 261. Preceding OpId from leader: term: 398 index: 262. (index mismatch)
I1014 19:26:52.986009 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.986141 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I1014 19:26:52.986232 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.986251 17409 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 397.261 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 260
I1014 19:26:52.986905 28640 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 260
I1014 19:26:52.987210 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.987392 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.987545 28640 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 397.261
I1014 19:26:52.987582 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:52.987628 28640 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 398 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:52.988119 17931 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.988353 17931 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.989101 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:52.989064 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.989239 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:52.989897 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.990047 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:52.990792 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 398 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:52.990963 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.090641 17931 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 398 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:53.090750 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:53.090813 17931 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 398 FOLLOWER]: Advancing to term 399
I1014 19:26:53.093287 17931 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.093665 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 398. Current term is 399. Ops: []
I1014 19:26:53.093866 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 399 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:53.094017 17931 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 399 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.094523 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 398 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:53.094594 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:53.094640 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 398 FOLLOWER]: Advancing to term 399
I1014 19:26:53.094565 18340 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 398.262, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I1014 19:26:53.094820 18340 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 LEADER]: Stepping down as leader of term 398
I1014 19:26:53.094856 18340 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:53.094821 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 399 candidate_status { last_received { term: 398 index: 262 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:53.095021 18340 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 398.262, Last appended by leader: 262, Current term: 398, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.095197 18340 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:53.095247 18340 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:53.095332 18340 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 398 FOLLOWER]: Advancing to term 399
I1014 19:26:53.096714 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.097120 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 399 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.097250 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 399 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.097450 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 399 candidate_status { last_received { term: 398 index: 262 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:53.097537 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (vote granted)
I1014 19:26:53.097617 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 399 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.097928 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 399 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 399 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.098127 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 399 candidate_status { last_received { term: 398 index: 262 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:53.098230 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 399 candidate_status { last_received { term: 398 index: 262 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:53.098381 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 399: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:53.098695 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 399 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 399: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:53.099400 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 399.
I1014 19:26:53.099535 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 399 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.099758 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 398. Current term is 399. Ops: []
I1014 19:26:53.099985 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 399 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 399 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.100051 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 399 election: Election decided. Result: candidate lost.
I1014 19:26:53.100129 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 399 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.100185 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 399 election: Election decided. Result: candidate won.
I1014 19:26:53.100409 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:26:53.100461 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Leader election lost for term 399. Reason: could not achieve majority
I1014 19:26:53.100628 17931 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:53.100721 17931 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Leader election won for term 399
I1014 19:26:53.100942 17931 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:53.101179 17931 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 262, Committed index: 262, Last appended: 398.262, Last appended by leader: 262, Current term: 399, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.102632 18463 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.102835 18463 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.186524 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 398 index: 262. Preceding OpId from leader: term: 399 index: 263. (index mismatch)
I1014 19:26:53.187019 17931 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I1014 19:26:53.188072 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.188279 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.189314 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.189540 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.189923 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 398 index: 262. Preceding OpId from leader: term: 399 index: 263. (index mismatch)
I1014 19:26:53.190001 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.190167 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.190593 18198 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 263, Last known committed idx: 262, Time since last communication: 0.000s
I1014 19:26:53.191751 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.191923 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.191902 18340 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.192085 18340 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.289836 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.290099 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.304388 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:53.305214 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:53.309165 12420 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:53.311168 12420 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:53.424216 12420 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:53.425159 12420 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:53.500175 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:53.501094 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:53.502462 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 395 vs 399: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:53.502820 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 399 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.530917 18340 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:53.530997 18340 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:53.531049 18340 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 399 FOLLOWER]: Advancing to term 400
I1014 19:26:53.533264 18340 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.533746 18340 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 400 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:53.533890 18340 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 400 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.534502 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 400 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:53.534732 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 400 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.535039 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 400 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:53.535218 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 400 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.535939 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 400 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 400 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.537048 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 400 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 400 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.537117 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 400 election: Election decided. Result: candidate lost.
I1014 19:26:53.537297 18340 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:53.537348 18340 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Leader election lost for term 400. Reason: could not achieve majority
I1014 19:26:53.540217 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 399. Current term is 400. Ops: []
I1014 19:26:53.540709 18463 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 399.265, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I1014 19:26:53.540966 18463 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 LEADER]: Stepping down as leader of term 399
I1014 19:26:53.541019 18463 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:53.541234 18463 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 399.265, Last appended by leader: 265, Current term: 399, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.541446 18463 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:53.541539 18463 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:53.541689 18463 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 399 FOLLOWER]: Advancing to term 400
I1014 19:26:53.621338 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:53.621423 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:53.621505 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 399 FOLLOWER]: Advancing to term 400
I1014 19:26:53.623733 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.624275 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 400 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.624434 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 400 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.624850 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 400 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:53.625035 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 400: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:53.625274 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 400 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:53.625453 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:53.625625 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 400 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 400: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:53.636682 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 400 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 400.
I1014 19:26:53.637110 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 400 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.637219 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 400 election: Election decided. Result: candidate won.
I1014 19:26:53.637500 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:26:53.637568 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 FOLLOWER]: Leader election won for term 400
I1014 19:26:53.637840 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:53.638041 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 399.265, Last appended by leader: 265, Current term: 400, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.638973 18732 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 400 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.639101 18732 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.639283 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:53.640909 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:53.667958 18340 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:53.668028 18340 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.184s (starting election)
I1014 19:26:53.668081 18340 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 400 FOLLOWER]: Advancing to term 401
I1014 19:26:53.669445 18340 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.669801 18340 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 401 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:53.669912 18340 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 401 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.670359 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 401 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:53.670527 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 401 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.670452 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 401 candidate_status { last_received { term: 399 index: 265 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:53.670634 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 400 FOLLOWER]: Advancing to term 401
I1014 19:26:53.670701 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (vote granted)
I1014 19:26:53.670843 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 401 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 401 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.672756 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 401 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 401.
I1014 19:26:53.673027 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 401 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.673074 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 401 election: Election decided. Result: candidate won.
I1014 19:26:53.673219 18340 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:26:53.673270 18340 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 FOLLOWER]: Leader election won for term 401
I1014 19:26:53.673349 18340 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:53.673519 18340 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 399.265, Last appended by leader: 265, Current term: 401, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.674238 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.674522 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.717659 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 401 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 400. Current term is 401. Ops: []
I1014 19:26:53.718147 18732 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I1014 19:26:53.718353 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 LEADER]: Stepping down as leader of term 400
I1014 19:26:53.718400 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:53.718580 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 400.266, Last appended by leader: 266, Current term: 400, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.718750 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:53.718780 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:53.718871 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 400 FOLLOWER]: Advancing to term 401
I1014 19:26:53.780984 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 401 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 400 index: 266. Preceding OpId from leader: term: 401 index: 266. (term mismatch)
I1014 19:26:53.781036 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 265
I1014 19:26:53.781056 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 400.266
I1014 19:26:53.781092 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 401 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:53.781839 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I1014 19:26:53.785132 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 401 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 399 index: 265. Preceding OpId from leader: term: 401 index: 266. (index mismatch)
I1014 19:26:53.785527 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.785722 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.785748 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I1014 19:26:53.786123 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.786289 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:53.786501 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.786665 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.787842 18463 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.788102 18463 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.788081 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 401 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:53.788259 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:53.887560 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 401 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:53.887650 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:53.887713 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 401 FOLLOWER]: Advancing to term 402
I1014 19:26:53.889969 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.890305 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 401. Current term is 402. Ops: []
I1014 19:26:53.890466 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 402 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.890830 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 402 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:53.891041 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 401 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:53.891161 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (starting election)
I1014 19:26:53.891211 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 401 FOLLOWER]: Advancing to term 402
I1014 19:26:53.893379 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.893857 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 402 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:53.894001 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 402 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:53.894589 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 402 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:53.894600 17409 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 401.266, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I1014 19:26:53.894698 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 402 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:53.894531 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 402 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:53.894985 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 402: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:53.895308 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 402 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 402: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:53.895507 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 402 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:53.895656 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 402: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:53.895985 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 402 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 402: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:53.906177 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 402 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.906409 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 402 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.906678 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 402 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 402 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:53.906750 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 402 election: Election decided. Result: candidate lost.
I1014 19:26:53.906810 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 401. Current term is 402. Ops: []
I1014 19:26:53.907140 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:26:53.907199 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Leader election lost for term 402. Reason: could not achieve majority
I1014 19:26:53.907251 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 402 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 402 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:53.907337 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 402 election: Election decided. Result: candidate lost.
I1014 19:26:53.907598 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:53.907656 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Leader election lost for term 402. Reason: could not achieve majority
I1014 19:26:53.907938 17409 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 401.266, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I1014 19:26:53.908090 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Stepping down as leader of term 401
I1014 19:26:53.908145 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:53.908322 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 401.266, Last appended by leader: 266, Current term: 401, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:53.908478 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:53.908516 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:53.908618 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 401 FOLLOWER]: Advancing to term 402
I1014 19:26:54.011557 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:54.011668 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:26:54.011723 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 402 FOLLOWER]: Advancing to term 403
I1014 19:26:54.013238 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.013664 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 403 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.013816 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 403 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.014490 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 403 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.014466 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 403 candidate_status { last_received { term: 401 index: 266 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.014642 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 402 FOLLOWER]: Advancing to term 403
I1014 19:26:54.014642 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 402 FOLLOWER]: Advancing to term 403
I1014 19:26:54.014710 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:54.014710 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (vote granted)
I1014 19:26:54.016811 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 403.
I1014 19:26:54.016856 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 403 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 403.
I1014 19:26:54.017242 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 403 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.017292 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 403 election: Election decided. Result: candidate won.
I1014 19:26:54.017356 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 403 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.017484 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:26:54.017536 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 FOLLOWER]: Leader election won for term 403
I1014 19:26:54.017609 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:54.017768 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 401.266, Last appended by leader: 266, Current term: 403, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.018457 18732 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.018635 18732 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.102830 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 401 index: 266. Preceding OpId from leader: term: 403 index: 267. (index mismatch)
I1014 19:26:54.103369 18732 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I1014 19:26:54.104472 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.104672 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.105728 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.105917 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.106199 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 403 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 401 index: 266. Preceding OpId from leader: term: 403 index: 267. (index mismatch)
I1014 19:26:54.106495 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.106669 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.106819 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I1014 19:26:54.108150 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.108422 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.108263 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.108568 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.208914 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 403 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:54.208984 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:26:54.209035 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 403 FOLLOWER]: Advancing to term 404
I1014 19:26:54.210412 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.210788 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 404 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.210954 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 404 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.211485 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 404 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.211685 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 404 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.211766 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 403 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.211969 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.211951 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 404 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.212122 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 404 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.212658 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 403. Current term is 404. Ops: []
I1014 19:26:54.212687 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 404 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 404 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.212905 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 404 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 404 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.212965 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 404 election: Election decided. Result: candidate lost.
I1014 19:26:54.213119 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:26:54.213171 18862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Leader election lost for term 404. Reason: could not achieve majority
I1014 19:26:54.213346 18732 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 403.267, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I1014 19:26:54.213501 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 LEADER]: Stepping down as leader of term 403
I1014 19:26:54.213538 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:54.213721 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 403.267, Last appended by leader: 267, Current term: 403, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.213840 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:54.213861 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:54.213937 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 403 FOLLOWER]: Advancing to term 404
I1014 19:26:54.298053 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:54.298147 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:54.298198 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 403 FOLLOWER]: Advancing to term 404
I1014 19:26:54.300055 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.300611 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 404 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.300778 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 404 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.301417 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 404 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.301609 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (vote granted)
I1014 19:26:54.301631 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 404 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.301826 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 404: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:54.302171 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 404 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 404: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:54.303733 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 404 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 404.
I1014 19:26:54.304190 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 404 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.304252 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 404 election: Election decided. Result: candidate won.
I1014 19:26:54.304487 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:26:54.304544 18198 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 FOLLOWER]: Leader election won for term 404
I1014 19:26:54.311811 18198 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:54.312063 18198 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 403.267, Last appended by leader: 267, Current term: 404, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.313561 18992 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 404 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.313956 18992 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.320480 18993 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:54.322851 18993 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:54.348645 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:54.348722 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:54.348770 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 404 FOLLOWER]: Advancing to term 405
I1014 19:26:54.351040 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.351536 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 405 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.351697 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 405 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.352258 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 405 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.352433 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 404 FOLLOWER]: Advancing to term 405
I1014 19:26:54.352396 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 405 candidate_status { last_received { term: 403 index: 267 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.352519 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (vote granted)
I1014 19:26:54.352617 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 405 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.353036 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 405 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 405 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.354629 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 405 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 405.
I1014 19:26:54.354950 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 405 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.355001 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 405 election: Election decided. Result: candidate won.
I1014 19:26:54.355274 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.223s (election complete)
I1014 19:26:54.355334 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 FOLLOWER]: Leader election won for term 405
I1014 19:26:54.355420 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:54.355628 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 403.267, Last appended by leader: 267, Current term: 405, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.356789 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.356920 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.425261 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 404. Current term is 405. Ops: []
I1014 19:26:54.425757 18992 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I1014 19:26:54.425935 18992 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 405.268 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 267
I1014 19:26:54.426000 18198 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 LEADER]: Stepping down as leader of term 404
I1014 19:26:54.426054 18198 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:54.426265 18198 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 404.268, Last appended by leader: 268, Current term: 404, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.426482 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:54.426537 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:54.426661 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 404 FOLLOWER]: Advancing to term 405
I1014 19:26:54.435041 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 405 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 404 index: 268. Preceding OpId from leader: term: 405 index: 268. (term mismatch)
I1014 19:26:54.435086 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 267
I1014 19:26:54.435107 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 404.268
I1014 19:26:54.435144 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 405 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:54.435685 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I1014 19:26:54.436739 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.436929 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.437657 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.437875 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.438560 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.438702 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.439003 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 405 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 403 index: 267. Preceding OpId from leader: term: 405 index: 268. (index mismatch)
I1014 19:26:54.439430 18998 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I1014 19:26:54.440704 18732 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.440886 18732 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.441068 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 405 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.441184 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.529582 29304 heartbeater.cc:467] Master 127.20.201.60:53788 was elected leader, sending a full tablet report...
I1014 19:26:54.539304 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 405 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:54.539391 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:54.539453 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 405 FOLLOWER]: Advancing to term 406
I1014 19:26:54.541765 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.542263 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 406 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.542414 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 406 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.543032 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 405 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:54.543020 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 406 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.543094 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:54.543134 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 405 FOLLOWER]: Advancing to term 406
I1014 19:26:54.543298 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 406 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.543439 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 406 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.543745 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 406 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 406 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.544811 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 406 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.545037 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 406 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 406: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:54.545120 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 406 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.545223 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 406 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.545580 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 406 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 406 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 406: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:54.545670 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 406 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.545687 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 406 election: Election decided. Result: candidate lost.
I1014 19:26:54.545812 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 406 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.545873 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 406 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.546033 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 406: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:54.546351 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 406 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 406 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.546607 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:54.546597 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 406 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 406: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:54.546658 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 406 election: Election decided. Result: candidate lost.
I1014 19:26:54.546658 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Leader election lost for term 406. Reason: could not achieve majority
I1014 19:26:54.546820 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:54.546854 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 406 FOLLOWER]: Leader election lost for term 406. Reason: could not achieve majority
I1014 19:26:54.548465 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 405. Current term is 406. Ops: []
I1014 19:26:54.549340 18998 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 405.268, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I1014 19:26:54.549549 18998 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Stepping down as leader of term 405
I1014 19:26:54.549589 18998 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:54.549764 18998 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 405.268, Last appended by leader: 268, Current term: 405, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.549896 18998 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:54.549932 18998 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:54.550027 18998 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 405 FOLLOWER]: Advancing to term 406
I1014 19:26:54.646230 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 406 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:54.646317 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:54.646369 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 406 FOLLOWER]: Advancing to term 407
I1014 19:26:54.648342 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.648886 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 407 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.649051 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 407 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.650085 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 407 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.650247 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 406 FOLLOWER]: Advancing to term 407
I1014 19:26:54.650313 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:54.650329 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 407 candidate_status { last_received { term: 405 index: 268 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.650511 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 406 FOLLOWER]: Advancing to term 407
I1014 19:26:54.650578 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.156s (vote granted)
I1014 19:26:54.652477 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 407 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 407.
I1014 19:26:54.652675 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 407 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 407.
I1014 19:26:54.652861 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 407 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.652918 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 407 election: Election decided. Result: candidate won.
I1014 19:26:54.653043 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 407 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.653120 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:54.653182 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 FOLLOWER]: Leader election won for term 407
I1014 19:26:54.653273 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:54.653456 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 405.268, Last appended by leader: 268, Current term: 407, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.654606 18998 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.654776 18998 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:54.712589 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.723s	user 0.002s	sys 0.000s
I1014 19:26:54.712632 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:54.713310 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:54.715183 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 407 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 405 index: 268. Preceding OpId from leader: term: 407 index: 270. (index mismatch)
I1014 19:26:54.715217 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 407 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 405 index: 268. Preceding OpId from leader: term: 407 index: 270. (index mismatch)
I1014 19:26:54.715647 18998 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I1014 19:26:54.715955 18862 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I1014 19:26:54.717425 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.717502 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.717643 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.717671 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.718394 17279 mvcc.cc:203] Tried to move safe_time back from 6306080418668830720 to 6306080418422718464. Current Snapshot: MvccSnapshot[committed={T|T < 6306080418668830720}]
I1014 19:26:54.718912 18998 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.719086 18998 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.719257 18998 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.719434 18998 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:54.722978 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.723141 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.723898 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:54.724642 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 407 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.724817 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:54.725102 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:54.819355 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 407 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:54.819428 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:54.819499 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 407 FOLLOWER]: Advancing to term 408
I1014 19:26:54.819641 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 407 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:54.819712 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:54.819784 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 407 FOLLOWER]: Advancing to term 408
I1014 19:26:54.821167 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 408 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.821655 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 408 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.821813 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 408 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.822012 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.822343 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 408 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.822348 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 408 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.822463 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 408 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.822491 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 408: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:54.822557 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 408 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.822630 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 408 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:54.822835 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 408 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 408: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:54.822993 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 408 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 408 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.823052 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 408 election: Election decided. Result: candidate lost.
I1014 19:26:54.823137 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 408 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.823276 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 408 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 408: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:54.823328 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:54.823377 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 408 FOLLOWER]: Leader election lost for term 408. Reason: could not achieve majority
I1014 19:26:54.823465 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 408 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:54.823602 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 408 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.823613 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 408 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 408 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 408: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:54.823856 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 408 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 408 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:54.823909 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 408 election: Election decided. Result: candidate lost.
I1014 19:26:54.824082 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:54.824134 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Leader election lost for term 408. Reason: could not achieve majority
I1014 19:26:54.825704 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 407. Current term is 408. Ops: []
I1014 19:26:54.826094 17409 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 407.270, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I1014 19:26:54.826311 18998 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Stepping down as leader of term 407
I1014 19:26:54.826354 18998 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:54.826611 18998 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 407.270, Last appended by leader: 270, Current term: 407, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.826747 18998 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:54.826784 18998 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:54.826867 18998 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 407 FOLLOWER]: Advancing to term 408
I1014 19:26:54.906698 18998 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 408 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:54.906769 18998 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:26:54.906821 18998 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 408 FOLLOWER]: Advancing to term 409
I1014 19:26:54.908910 18998 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.909363 18998 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 409 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.909524 18998 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 409 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.910437 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 409 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:54.910586 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 408 FOLLOWER]: Advancing to term 409
I1014 19:26:54.910571 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 409 candidate_status { last_received { term: 407 index: 270 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:54.910650 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.176s (vote granted)
I1014 19:26:54.910706 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 408 FOLLOWER]: Advancing to term 409
I1014 19:26:54.910769 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.201s (vote granted)
I1014 19:26:54.912778 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 409 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 409.
I1014 19:26:54.912780 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 409.
I1014 19:26:54.913116 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 409 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:54.913167 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 409 election: Election decided. Result: candidate won.
I1014 19:26:54.913225 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 409 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:54.913421 18998 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:26:54.913475 18998 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 FOLLOWER]: Leader election won for term 409
I1014 19:26:54.913569 18998 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:54.913738 18998 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 407.270, Last appended by leader: 270, Current term: 409, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:54.914708 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:54.914897 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:26:54.995813 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.355s	user 0.002s	sys 0.000s
I1014 19:26:54.995872 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:54.996822 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:54.997929 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 400 vs 409: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 409 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:54.998190 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 409 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:54.998425 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 409 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:54.998667 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 409 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.007375 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 409 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 407 index: 270. Preceding OpId from leader: term: 409 index: 271. (index mismatch)
I1014 19:26:55.007962 17409 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I1014 19:26:55.009244 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.009464 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.010699 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.010923 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.011479 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 407 index: 270. Preceding OpId from leader: term: 409 index: 271. (index mismatch)
I1014 19:26:55.011955 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.012133 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.012228 18998 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I1014 19:26:55.014982 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.015241 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.015339 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.015619 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.099022 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 409 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.099262 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.112361 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 409 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:55.112473 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:26:55.112537 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 409 FOLLOWER]: Advancing to term 410
I1014 19:26:55.114815 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.115319 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 410 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.115510 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 410 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.116255 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 410 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.116271 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 410 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.116411 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 410 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.116470 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 410 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.116788 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 410 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 410 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.116932 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 410 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 410 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.117000 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 410 election: Election decided. Result: candidate lost.
I1014 19:26:55.117480 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:55.117560 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Leader election lost for term 410. Reason: could not achieve majority
I1014 19:26:55.131237 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 409. Current term is 410. Ops: []
I1014 19:26:55.131721 18862 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 409.271, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I1014 19:26:55.131907 17409 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 LEADER]: Stepping down as leader of term 409
I1014 19:26:55.131963 17409 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:55.132133 17409 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 409.271, Last appended by leader: 271, Current term: 409, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.132314 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:55.132361 17409 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:55.132511 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 409 FOLLOWER]: Advancing to term 410
I1014 19:26:55.200058 19256 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:55.200136 19256 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:55.200186 19256 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 409 FOLLOWER]: Advancing to term 410
I1014 19:26:55.202250 19256 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.202675 19256 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 410 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.202823 19256 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 410 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.203282 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 410 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.203420 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:55.203373 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 410 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.203583 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 410: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:55.203994 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 410 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 410: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:55.205031 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 410 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 410.
I1014 19:26:55.205317 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 410 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.205365 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 410 election: Election decided. Result: candidate won.
I1014 19:26:55.205607 19256 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:26:55.205662 19256 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 FOLLOWER]: Leader election won for term 410
I1014 19:26:55.205760 19256 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.205968 19256 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 409.271, Last appended by leader: 271, Current term: 410, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.206817 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 410 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.206986 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.207146 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:55.208758 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:55.255694 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.255772 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (starting election)
I1014 19:26:55.255823 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 410 FOLLOWER]: Advancing to term 411
I1014 19:26:55.258036 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.258523 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 411 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.258677 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 411 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.259239 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 411 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.259238 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 411 candidate_status { last_received { term: 409 index: 271 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.259393 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 410 FOLLOWER]: Advancing to term 411
I1014 19:26:55.259446 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 411 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.259461 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (vote granted)
I1014 19:26:55.260465 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 411 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 411 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.260988 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 411 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 411.
I1014 19:26:55.261272 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 411 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.261335 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 411 election: Election decided. Result: candidate won.
I1014 19:26:55.261627 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.202s (election complete)
I1014 19:26:55.261684 18198 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 FOLLOWER]: Leader election won for term 411
I1014 19:26:55.261993 18198 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:55.262182 18198 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 409.271, Last appended by leader: 271, Current term: 411, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.263347 19370 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.263558 19370 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.308166 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 410. Current term is 411. Ops: []
I1014 19:26:55.308643 17537 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I1014 19:26:55.308811 17537 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 411.272 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 271
I1014 19:26:55.308876 19256 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 LEADER]: Stepping down as leader of term 410
I1014 19:26:55.308923 19256 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.309087 19256 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 410.272, Last appended by leader: 272, Current term: 410, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.309271 19256 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:55.309305 19256 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:55.309403 19256 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 410 FOLLOWER]: Advancing to term 411
I1014 19:26:55.372982 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 411 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 409 index: 271. Preceding OpId from leader: term: 411 index: 272. (index mismatch)
I1014 19:26:55.375304 19370 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I1014 19:26:55.377176 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.377362 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.379977 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.380187 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.380697 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 410 index: 272. Preceding OpId from leader: term: 411 index: 272. (term mismatch)
I1014 19:26:55.380746 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 271
I1014 19:26:55.380764 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 410.272
I1014 19:26:55.380795 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:55.383198 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.383404 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.383862 18198 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s
I1014 19:26:55.385568 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.385769 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.385779 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.385956 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.470844 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 411 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.471076 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.482122 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 411 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:55.482197 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:26:55.482244 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 411 FOLLOWER]: Advancing to term 412
I1014 19:26:55.484429 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.484933 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 412 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.485075 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 412 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.485551 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 412 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.485756 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 412 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.485747 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 411. Current term is 412. Ops: []
I1014 19:26:55.486038 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 412 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.486233 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 412 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.486577 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 412 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 412 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.486773 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 412 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 412 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.486836 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 412 election: Election decided. Result: candidate lost.
I1014 19:26:55.487015 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (election complete)
I1014 19:26:55.487095 17409 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Leader election lost for term 412. Reason: could not achieve majority
I1014 19:26:55.487321 19370 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 411.272, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:55.487562 18198 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Stepping down as leader of term 411
I1014 19:26:55.487604 18198 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:55.487776 18198 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 411.272, Last appended by leader: 272, Current term: 411, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.487951 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:55.487992 18198 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:55.488111 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 411 FOLLOWER]: Advancing to term 412
I1014 19:26:55.571797 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:55.571887 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:26:55.571943 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 411 FOLLOWER]: Advancing to term 412
I1014 19:26:55.573462 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.573864 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 412 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.574023 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 412 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.574625 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 412 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.574625 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 412 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.574774 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (vote granted)
I1014 19:26:55.574807 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 412: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:55.575134 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 412 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 412: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:55.576576 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 412 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 412.
I1014 19:26:55.576912 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 412 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.576962 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 412 election: Election decided. Result: candidate won.
I1014 19:26:55.577153 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:55.577208 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 FOLLOWER]: Leader election won for term 412
I1014 19:26:55.577289 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.577471 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 411.272, Last appended by leader: 272, Current term: 412, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.578161 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 412 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.578382 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.618727 17409 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.618813 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.204s (starting election)
I1014 19:26:55.618872 17409 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 412 FOLLOWER]: Advancing to term 413
I1014 19:26:55.621095 17409 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.621626 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 413 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.621829 17409 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 413 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.622292 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 413 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.622512 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 413 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.622454 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 413 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.622632 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 412 FOLLOWER]: Advancing to term 413
I1014 19:26:55.622700 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:55.623075 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 413 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 413 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.624274 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 413 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 413.
I1014 19:26:55.624575 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 413 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.624626 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 413 election: Election decided. Result: candidate won.
I1014 19:26:55.624769 17409 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:55.624819 17409 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 FOLLOWER]: Leader election won for term 413
I1014 19:26:55.624902 17409 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:55.625109 17409 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 411.272, Last appended by leader: 272, Current term: 413, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.626085 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 413 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.626271 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.660519 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 413 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 412. Current term is 413. Ops: []
I1014 19:26:55.661069 19256 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:55.661298 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 LEADER]: Stepping down as leader of term 412
I1014 19:26:55.661350 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.661559 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 412.273, Last appended by leader: 273, Current term: 412, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.661739 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:55.661767 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:55.661859 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 412 FOLLOWER]: Advancing to term 413
W1014 19:26:55.708822 18993 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.386s	user 0.001s	sys 0.000s
I1014 19:26:55.708878 18993 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:55.709818 18993 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:55.711367 18993 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 404 vs 413: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 413 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:55.711804 18993 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 413 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:55.712224 18993 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 413 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:55.712555 18993 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 413 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.726193 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 413 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.726310 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:26:55.726370 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 413 FOLLOWER]: Advancing to term 414
I1014 19:26:55.728680 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 414 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.729197 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 414 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.730429 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 414 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.730793 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 414 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.730967 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 413 FOLLOWER]: Advancing to term 414
I1014 19:26:55.731663 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 414 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.731847 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 414 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.732400 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 414 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 414 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.733134 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 414 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 414 because replica has last-logged OpId of term: 412 index: 273, which is greater than that of the candidate, which has last-logged OpId of term: 411 index: 272.
I1014 19:26:55.733489 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 414 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 414 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 414 because replica has last-logged OpId of term: 412 index: 273, which is greater than that of the candidate, which has last-logged OpId of term: 411 index: 272.
I1014 19:26:55.733595 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 414 election: Election decided. Result: candidate lost.
I1014 19:26:55.733944 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:26:55.734004 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 414 FOLLOWER]: Leader election lost for term 414. Reason: could not achieve majority
I1014 19:26:55.740707 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 414 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 413. Current term is 414. Ops: []
I1014 19:26:55.741230 18862 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:55.748050 18862 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 412.273 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 272
I1014 19:26:55.748409 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 LEADER]: Stepping down as leader of term 413
I1014 19:26:55.748762 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:55.749049 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 413.273, Last appended by leader: 273, Current term: 413, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.749284 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:55.749348 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:55.749544 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 413 FOLLOWER]: Advancing to term 414
I1014 19:26:55.752262 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 414 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 413. Current term is 414. Ops: []
I1014 19:26:55.756096 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 414 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.756166 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (starting election)
I1014 19:26:55.756214 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 414 FOLLOWER]: Advancing to term 415
I1014 19:26:55.758400 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.758911 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 415 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.759074 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 415 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.759567 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 415 candidate_status { last_received { term: 412 index: 273 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.759719 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 414 FOLLOWER]: Advancing to term 415
I1014 19:26:55.759675 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 415 candidate_status { last_received { term: 412 index: 273 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.759824 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 414 FOLLOWER]: Advancing to term 415
I1014 19:26:55.759888 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.166s (vote granted)
I1014 19:26:55.761881 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 415 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 415 because replica has last-logged OpId of term: 413 index: 273, which is greater than that of the candidate, which has last-logged OpId of term: 412 index: 273.
I1014 19:26:55.762060 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 415 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 415.
I1014 19:26:55.762203 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 415 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 415 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 415 because replica has last-logged OpId of term: 413 index: 273, which is greater than that of the candidate, which has last-logged OpId of term: 412 index: 273.
I1014 19:26:55.762373 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 415 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.762425 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 415 election: Election decided. Result: candidate won.
I1014 19:26:55.762610 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:55.762666 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 FOLLOWER]: Leader election won for term 415
I1014 19:26:55.762744 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.762922 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 412.273, Last appended by leader: 273, Current term: 415, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.763967 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 415 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.764153 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.868999 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 415 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.869072 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:26:55.869117 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 415 FOLLOWER]: Advancing to term 416
I1014 19:26:55.870805 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.871176 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 416 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.871312 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 416 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.871922 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 416 candidate_status { last_received { term: 413 index: 273 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.872109 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 416 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.872257 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 416 candidate_status { last_received { term: 413 index: 273 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:55.872440 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 415 FOLLOWER]: Advancing to term 416
I1014 19:26:55.872519 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:55.872556 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 416 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 416 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:55.874356 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 416 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 416.
I1014 19:26:55.874866 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 416 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:55.874912 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 416 election: Election decided. Result: candidate won.
I1014 19:26:55.875090 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:55.875178 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 FOLLOWER]: Leader election won for term 416
I1014 19:26:55.875314 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:55.875551 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 413.273, Last appended by leader: 273, Current term: 416, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.876726 19517 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.876996 19517 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:55.877178 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 415. Current term is 416. Ops: []
I1014 19:26:55.877624 19256 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 274, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:55.877770 19256 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 416.274 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 272
I1014 19:26:55.877861 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 LEADER]: Stepping down as leader of term 415
I1014 19:26:55.877907 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:55.878057 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 415.274, Last appended by leader: 274, Current term: 415, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.878183 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:55.878221 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:55.878300 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 415 FOLLOWER]: Advancing to term 416
I1014 19:26:55.983476 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 415 index: 274. Preceding OpId from leader: term: 416 index: 274. (term mismatch)
I1014 19:26:55.983536 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 273
I1014 19:26:55.983551 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 415.274
I1014 19:26:55.983587 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:55.984197 19517 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:55.984364 19517 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 412.273 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 272
I1014 19:26:55.985159 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 272
I1014 19:26:55.985185 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 412.273
I1014 19:26:55.985215 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:55.985779 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:55.985965 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:55.989382 18198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 416 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:55.989455 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:55.989528 18198 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 416 FOLLOWER]: Advancing to term 417
I1014 19:26:55.991683 18198 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:55.992225 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 417 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:55.992411 18198 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 417 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:55.992841 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 417 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:55.993072 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 417 candidate_status { last_received { term: 411 index: 272 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:55.998589 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 416. Current term is 417. Ops: []
I1014 19:26:55.999250 18862 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 274, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:56.000933 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 417 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.001049 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Stepping down as leader of term 416
I1014 19:26:56.001157 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:56.001400 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 416.274, Last appended by leader: 274, Current term: 416, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.001618 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:56.001675 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:56.001705 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 417 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 417 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.001727 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 416 FOLLOWER]: Advancing to term 417
I1014 19:26:56.004168 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 416. Current term is 417. Ops: []
I1014 19:26:56.004664 19517 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 417 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.004882 19517 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.005228 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 417 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:56.005203 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.005414 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.005560 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.005699 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 417 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 417 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:56.005741 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.005772 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 417 election: Election decided. Result: candidate lost.
I1014 19:26:56.005981 18198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.178s (election complete)
I1014 19:26:56.006038 18198 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Leader election lost for term 417. Reason: could not achieve majority
I1014 19:26:56.040726 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:56.041636 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:56.043067 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 410 vs 416: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.043357 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.043656 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 416 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.087538 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:26:56.087610 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:56.087662 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 416 FOLLOWER]: Advancing to term 417
I1014 19:26:56.089195 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.089619 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 417 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.089756 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 417 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.090306 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 417 candidate_status { last_received { term: 416 index: 274 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.090304 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 417 candidate_status { last_received { term: 416 index: 274 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.090513 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 417: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:56.090507 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:56.092298 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 417.
I1014 19:26:56.092641 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 417 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.092710 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 417 election: Election decided. Result: candidate won.
I1014 19:26:56.092767 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 417 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 417: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:56.092950 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:56.093004 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 FOLLOWER]: Leader election won for term 417
I1014 19:26:56.093080 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:56.093272 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 416.274, Last appended by leader: 274, Current term: 417, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.094360 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.094607 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.094776 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:56.096249 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:56.181814 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 411 index: 272. Preceding OpId from leader: term: 417 index: 275. (index mismatch)
I1014 19:26:56.182305 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 272, Time since last communication: 0.000s
I1014 19:26:56.184943 18198 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.185153 18198 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.185309 19651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.185554 19651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.185988 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.186178 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.185101 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.186606 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.186873 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 416 index: 274. Preceding OpId from leader: term: 417 index: 275. (index mismatch)
I1014 19:26:56.186408 19651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.187315 19651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.187280 19256 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I1014 19:26:56.188694 19517 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.188694 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.188993 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.188994 19517 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.276590 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 417 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.276825 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.286562 19651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:56.286648 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:26:56.286693 19651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 417 FOLLOWER]: Advancing to term 418
I1014 19:26:56.288893 19651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.289393 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 418 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.289592 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 418 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.290175 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 418 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.290364 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 418 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.290340 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 418 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.290556 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 418 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.290863 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 418 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 418 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.290997 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 418 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 418 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.291059 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 418 election: Election decided. Result: candidate lost.
I1014 19:26:56.291224 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:26:56.291307 19651 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Leader election lost for term 418. Reason: could not achieve majority
I1014 19:26:56.296396 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 417. Current term is 418. Ops: []
I1014 19:26:56.296829 19667 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 417.275, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.297067 19256 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 LEADER]: Stepping down as leader of term 417
I1014 19:26:56.297107 19256 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:56.297309 19256 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 417.275, Last appended by leader: 275, Current term: 417, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.297478 19256 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:56.297518 19256 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:56.297618 19256 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 417 FOLLOWER]: Advancing to term 418
I1014 19:26:56.378195 19517 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:56.378345 19517 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:56.378446 19517 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 417 FOLLOWER]: Advancing to term 418
I1014 19:26:56.380818 19517 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.385674 19517 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 418 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.386302 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 418 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.386454 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (vote granted)
I1014 19:26:56.388581 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 418 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 418.
I1014 19:26:56.389062 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 418 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.389132 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 418 election: Election decided. Result: candidate won.
I1014 19:26:56.389451 19517 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 418 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.389451 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:26:56.389616 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 FOLLOWER]: Leader election won for term 418
I1014 19:26:56.389786 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:56.390008 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 417.275, Last appended by leader: 275, Current term: 418, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.390161 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 418 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.390338 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 418: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:56.390923 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 418 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 418: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:56.391618 17409 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 418 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.391825 17409 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.406354 19651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:56.406435 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:26:56.406503 19651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 418 FOLLOWER]: Advancing to term 419
I1014 19:26:56.408792 19651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.409327 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 419 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.410028 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 419 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.410181 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 418 FOLLOWER]: Advancing to term 419
I1014 19:26:56.410251 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (vote granted)
I1014 19:26:56.410431 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 419 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.411015 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 419 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.411203 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 419 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.411645 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 419 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 419 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.412269 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 419 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 419.
I1014 19:26:56.412546 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 419 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.412600 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 419 election: Election decided. Result: candidate won.
I1014 19:26:56.412940 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.224s (election complete)
I1014 19:26:56.413029 19651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 FOLLOWER]: Leader election won for term 419
I1014 19:26:56.413177 19651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:56.413496 19651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 417.275, Last appended by leader: 275, Current term: 419, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.426249 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 419 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.426496 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.435294 19783 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:56.439944 19783 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:56.491680 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 419 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 418. Current term is 419. Ops: []
I1014 19:26:56.492398 18862 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.492707 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 LEADER]: Stepping down as leader of term 418
I1014 19:26:56.492796 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:56.493001 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 418.276, Last appended by leader: 276, Current term: 418, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.493202 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:56.493254 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:56.493366 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 418. Current term is 419. Ops: []
I1014 19:26:56.493403 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 418 FOLLOWER]: Advancing to term 419
W1014 19:26:56.507028 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.782s	user 0.001s	sys 0.000s
I1014 19:26:56.507089 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:56.508008 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:56.509183 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 407 vs 419: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.509568 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.509873 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.510138 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.510365 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.510654 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.510949 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.511191 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.511494 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.511721 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.511996 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.512231 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.512462 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.512714 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.512941 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.513217 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:56.513458 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 419 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.522166 19256 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 419 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:56.522241 19256 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:26:56.522294 19256 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 419 FOLLOWER]: Advancing to term 420
I1014 19:26:56.524457 19256 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 420 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.524878 19256 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 420 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.525020 19256 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 420 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.525718 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 420 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.525718 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 420 candidate_status { last_received { term: 417 index: 275 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.525904 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 420 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.525904 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 419 FOLLOWER]: Advancing to term 420
I1014 19:26:56.526324 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 420 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 420 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.527901 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 420 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 420 because replica has last-logged OpId of term: 418 index: 276, which is greater than that of the candidate, which has last-logged OpId of term: 417 index: 275.
I1014 19:26:56.528280 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 420 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 420 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 420 because replica has last-logged OpId of term: 418 index: 276, which is greater than that of the candidate, which has last-logged OpId of term: 417 index: 275.
I1014 19:26:56.528340 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 420 election: Election decided. Result: candidate lost.
I1014 19:26:56.528393 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 420 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 419. Current term is 420. Ops: []
I1014 19:26:56.528657 19256 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:26:56.528717 19256 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 420 FOLLOWER]: Leader election lost for term 420. Reason: could not achieve majority
I1014 19:26:56.531201 19652 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.531389 19652 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 418.276 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 275
I1014 19:26:56.531455 19651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Stepping down as leader of term 419
I1014 19:26:56.531524 19651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:56.531735 19651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 419.276, Last appended by leader: 276, Current term: 419, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.531924 19651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:56.531963 19651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:56.532150 19651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 419 FOLLOWER]: Advancing to term 420
I1014 19:26:56.538239 19783 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:56.539208 19783 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:56.541656 19783 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 419 vs 420: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 420 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.594604 19517 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 420 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:56.594717 19517 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (starting election)
I1014 19:26:56.594799 19517 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 420 FOLLOWER]: Advancing to term 421
I1014 19:26:56.597304 19517 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.597848 19517 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 421 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.598063 19517 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 421 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.598561 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 421 candidate_status { last_received { term: 418 index: 276 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.598719 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 420 FOLLOWER]: Advancing to term 421
I1014 19:26:56.598801 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:56.598698 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 421 candidate_status { last_received { term: 418 index: 276 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.598912 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 420 FOLLOWER]: Advancing to term 421
I1014 19:26:56.600977 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 421 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 421.
I1014 19:26:56.601240 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 421 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 421 because replica has last-logged OpId of term: 419 index: 276, which is greater than that of the candidate, which has last-logged OpId of term: 418 index: 276.
I1014 19:26:56.601425 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 421 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.601496 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 421 election: Election decided. Result: candidate won.
I1014 19:26:56.601703 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 421 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 421 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 421 because replica has last-logged OpId of term: 419 index: 276, which is greater than that of the candidate, which has last-logged OpId of term: 418 index: 276.
I1014 19:26:56.601807 19517 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:56.601954 19517 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 FOLLOWER]: Leader election won for term 421
I1014 19:26:56.602108 19517 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:56.602373 19517 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 418.276, Last appended by leader: 276, Current term: 421, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.603317 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 421 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.603561 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.603796 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:56.605234 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:56.611610 19651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 421 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:56.611678 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (starting election)
I1014 19:26:56.611734 19651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 421 FOLLOWER]: Advancing to term 422
I1014 19:26:56.613142 19651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.613575 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 422 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.613755 19651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 422 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.614181 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 422 candidate_status { last_received { term: 419 index: 276 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.614322 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 421 FOLLOWER]: Advancing to term 422
I1014 19:26:56.614311 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 422 candidate_status { last_received { term: 419 index: 276 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.614395 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (vote granted)
I1014 19:26:56.614472 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 422 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.615113 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 422 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 422 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.616364 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 422 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 422.
I1014 19:26:56.616694 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 422 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.616747 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 422 election: Election decided. Result: candidate won.
I1014 19:26:56.616950 19651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:56.617033 19651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 FOLLOWER]: Leader election won for term 422
I1014 19:26:56.617146 19651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:56.617403 19651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 419.276, Last appended by leader: 276, Current term: 422, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.618654 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.618830 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.618978 19783 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:56.620293 19783 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:56.679208 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 422 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 421. Current term is 422. Ops: []
I1014 19:26:56.679733 19517 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 277, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.679958 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 LEADER]: Stepping down as leader of term 421
I1014 19:26:56.680016 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:56.680266 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 421.277, Last appended by leader: 277, Current term: 421, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.680471 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:56.680528 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:56.680634 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 421 FOLLOWER]: Advancing to term 422
I1014 19:26:56.717156 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 422 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 421 index: 277. Preceding OpId from leader: term: 422 index: 277. (term mismatch)
I1014 19:26:56.717224 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 276
I1014 19:26:56.717258 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 421.277
I1014 19:26:56.717309 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 422 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:56.717819 19652 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.717970 19652 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 418.276 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 275
I1014 19:26:56.718629 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 275
I1014 19:26:56.718665 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 418.276
I1014 19:26:56.718708 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 422 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:56.719489 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.719727 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.720984 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.721194 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.721434 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 422 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 417 index: 275. Preceding OpId from leader: term: 422 index: 277. (index mismatch)
I1014 19:26:56.721657 19517 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.721839 19517 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.721844 19652 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 275, Time since last communication: 0.000s
I1014 19:26:56.722009 19517 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.722201 19517 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.723739 19256 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.723925 19256 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.723538 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.724046 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.723646 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.724181 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:56.821642 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 422 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:56.821736 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (starting election)
I1014 19:26:56.821805 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 422 FOLLOWER]: Advancing to term 423
I1014 19:26:56.822938 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 422 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:56.823035 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:56.823081 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 422 FOLLOWER]: Advancing to term 423
I1014 19:26:56.823701 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.824187 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 423 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.824354 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 423 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.825178 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 423 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.825179 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 423 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.825359 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 423 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.825237 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.825613 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 423: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:56.825772 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 423 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.825901 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 423 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:56.826000 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 423 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 423 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.826246 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 423 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 423: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:56.826377 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 423 election: Election decided. Result: candidate lost.
I1014 19:26:56.826503 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 423 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.826658 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:26:56.826730 18862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Leader election lost for term 423. Reason: could not achieve majority
I1014 19:26:56.826869 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 423: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:56.827245 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 423 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 423: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:56.828969 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 423 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:56.829108 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 423 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.829365 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 423 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 423 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:56.829464 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 423 election: Election decided. Result: candidate lost.
I1014 19:26:56.829710 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:56.829766 17537 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Leader election lost for term 423. Reason: could not achieve majority
I1014 19:26:56.830471 19651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 422 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.830729 19651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.831303 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 422. Current term is 423. Ops: []
I1014 19:26:56.831864 19651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 422.277, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I1014 19:26:56.835171 19652 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Stepping down as leader of term 422
I1014 19:26:56.835212 19652 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:56.835386 19652 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 422.277, Last appended by leader: 277, Current term: 422, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.835605 19652 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:56.835649 19652 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:56.835749 19652 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 422 FOLLOWER]: Advancing to term 423
I1014 19:26:56.835918 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 422. Current term is 423. Ops: []
I1014 19:26:56.931490 19652 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 423 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:56.931572 19652 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:26:56.931627 19652 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 423 FOLLOWER]: Advancing to term 424
I1014 19:26:56.934002 19652 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.934538 19652 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 424 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:56.935673 19652 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 424 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.936362 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 424 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:56.936542 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 423 FOLLOWER]: Advancing to term 424
I1014 19:26:56.936626 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.214s (vote granted)
I1014 19:26:56.938799 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 424 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 424.
I1014 19:26:56.939185 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 424 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:56.939256 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 424 election: Election decided. Result: candidate won.
I1014 19:26:56.939608 19652 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:56.939581 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 424 candidate_status { last_received { term: 422 index: 277 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:56.939690 19652 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 FOLLOWER]: Leader election won for term 424
I1014 19:26:56.939725 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 423 FOLLOWER]: Advancing to term 424
I1014 19:26:56.939774 19652 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:56.939790 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.193s (vote granted)
I1014 19:26:56.939990 19652 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 422.277, Last appended by leader: 277, Current term: 424, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:56.940946 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:56.941217 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:56.941720 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 424.
I1014 19:26:56.942121 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 424 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.034080 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 424 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 422 index: 277. Preceding OpId from leader: term: 424 index: 278. (index mismatch)
I1014 19:26:57.034613 19652 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I1014 19:26:57.035897 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.036190 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.036664 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 422 index: 277. Preceding OpId from leader: term: 424 index: 278. (index mismatch)
I1014 19:26:57.037210 19651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I1014 19:26:57.037780 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.038079 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.039104 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.039290 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.039335 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.039542 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.039441 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.039631 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.131649 19651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 424 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.131821 19651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.139081 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 424 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:57.139181 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:26:57.139267 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 424 FOLLOWER]: Advancing to term 425
I1014 19:26:57.140960 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 425 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.141403 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 425 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.141592 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 425 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.142143 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 425 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.142307 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.142612 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 425 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.142796 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.142782 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 425 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.143147 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 425 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.143223 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 425 election: Election decided. Result: candidate lost.
I1014 19:26:57.143462 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (election complete)
I1014 19:26:57.143533 18862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 425 FOLLOWER]: Leader election lost for term 425. Reason: could not achieve majority
I1014 19:26:57.148262 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 425 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 424. Current term is 425. Ops: []
I1014 19:26:57.148718 19651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 424.278, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I1014 19:26:57.148946 19651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 LEADER]: Stepping down as leader of term 424
I1014 19:26:57.148984 19651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:57.149164 19651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 424.278, Last appended by leader: 278, Current term: 424, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.149380 19651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:57.149421 19651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:57.149567 19651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 424 FOLLOWER]: Advancing to term 425
I1014 19:26:57.229362 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 425 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.229477 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:26:57.229568 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 425 FOLLOWER]: Advancing to term 426
I1014 19:26:57.232048 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.232746 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:57.232831 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:57.232884 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 424 FOLLOWER]: Advancing to term 425
I1014 19:26:57.232898 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 426 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.233099 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 426 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.233808 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 426 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.233793 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 426 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.233990 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 425 FOLLOWER]: Advancing to term 426
I1014 19:26:57.234073 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.155s (vote granted)
I1014 19:26:57.235198 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 425 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.235474 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 425 FOLLOWER]: Advancing to term 426
I1014 19:26:57.235544 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (vote granted)
I1014 19:26:57.235744 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 425 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.236255 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 426.
I1014 19:26:57.236382 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 425 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.236574 28826 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 425. Current term is 426.
W1014 19:26:57.237015 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 425 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 425. Current term is 426.
I1014 19:26:57.237046 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 425 election: Cancelling election due to peer responding with higher term
I1014 19:26:57.237053 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 426 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.237103 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 426 election: Election decided. Result: candidate won.
I1014 19:26:57.237308 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:26:57.237321 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 425 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.237393 19920 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 FOLLOWER]: Leader election won for term 426
I1014 19:26:57.237615 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 426.
I1014 19:26:57.237627 19920 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:57.237874 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:26:57.237937 19667 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Leader election lost for term 425. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 425. Current term is 426.
I1014 19:26:57.238221 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 426 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.237893 19920 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 278, Committed index: 278, Last appended: 424.278, Last appended by leader: 278, Current term: 426, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.238718 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 425 candidate_status { last_received { term: 424 index: 278 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.239490 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.239444 19651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.239683 19651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:57.240367 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 425 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 425 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.317682 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 424 index: 278. Preceding OpId from leader: term: 426 index: 279. (index mismatch)
I1014 19:26:57.318197 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I1014 19:26:57.319555 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.319795 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.320592 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.320839 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.321482 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.321736 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.321755 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 424 index: 278. Preceding OpId from leader: term: 426 index: 279. (index mismatch)
I1014 19:26:57.322516 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 279, Last known committed idx: 278, Time since last communication: 0.000s
I1014 19:26:57.323715 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.323938 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.323832 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.324126 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:57.407368 19652 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 426 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.407697 19652 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.421722 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:57.421818 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:26:57.421887 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 426 FOLLOWER]: Advancing to term 427
I1014 19:26:57.423593 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.424015 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 427 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.424180 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 427 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.424765 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 427 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.424921 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 427 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.424839 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 427 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.425056 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 427 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.425266 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 427 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 427 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.425544 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 427 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 427 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.425612 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 427 election: Election decided. Result: candidate lost.
I1014 19:26:57.425822 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:26:57.425901 18862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Leader election lost for term 427. Reason: could not achieve majority
I1014 19:26:57.441905 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 426. Current term is 427. Ops: []
I1014 19:26:57.442404 19652 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 426.279, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:57.442649 19920 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Stepping down as leader of term 426
I1014 19:26:57.442690 19920 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:57.442873 19920 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 426.279, Last appended by leader: 279, Current term: 426, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.443054 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:57.443097 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:57.443212 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 426 FOLLOWER]: Advancing to term 427
I1014 19:26:57.509573 17537 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:57.509654 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:26:57.509704 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 426 FOLLOWER]: Advancing to term 427
I1014 19:26:57.511304 17537 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.511780 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 427 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.511930 17537 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 427 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.512528 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 427 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.512763 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 427: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:57.512779 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 427 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.512938 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (vote granted)
I1014 19:26:57.515033 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 427 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 427.
I1014 19:26:57.515542 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 427 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 427: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:57.516305 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 427 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.516350 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 427 election: Election decided. Result: candidate won.
I1014 19:26:57.516666 17537 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:26:57.516719 17537 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 FOLLOWER]: Leader election won for term 427
I1014 19:26:57.516788 17537 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:57.516952 17537 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 426.279, Last appended by leader: 279, Current term: 427, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.518270 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 427 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.518424 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.544345 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.544430 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.216s (starting election)
I1014 19:26:57.544530 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 427 FOLLOWER]: Advancing to term 428
I1014 19:26:57.546561 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.547068 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 428 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.547230 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 428 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.548586 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 428 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.548568 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 428 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.548764 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 428 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.548764 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 427 FOLLOWER]: Advancing to term 428
I1014 19:26:57.548844 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:26:57.549217 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 428 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 428 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.551048 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 428 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 428.
I1014 19:26:57.551492 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 428 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.551584 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 428 election: Election decided. Result: candidate won.
I1014 19:26:57.588590 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:26:57.588691 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 FOLLOWER]: Leader election won for term 428
I1014 19:26:57.589584 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:57.589912 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 426.279, Last appended by leader: 279, Current term: 428, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.591094 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 428 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.591682 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.606530 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 427. Current term is 428. Ops: []
I1014 19:26:57.607134 17537 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:57.607338 17537 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 428.280 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 279
I1014 19:26:57.607421 17537 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 LEADER]: Stepping down as leader of term 427
I1014 19:26:57.607455 17537 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:57.607657 17537 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 427.280, Last appended by leader: 280, Current term: 427, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.607858 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:57.607893 17537 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:57.608009 17537 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 427 FOLLOWER]: Advancing to term 428
I1014 19:26:57.674090 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 428 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.674180 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:26:57.674232 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 428 FOLLOWER]: Advancing to term 429
I1014 19:26:57.676568 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 429 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.677187 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 429 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.677603 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 429 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.678217 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 429 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.678258 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 429 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.678422 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 429 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.678409 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 428 FOLLOWER]: Advancing to term 429
I1014 19:26:57.680750 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 429 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 429 because replica has last-logged OpId of term: 427 index: 280, which is greater than that of the candidate, which has last-logged OpId of term: 426 index: 279.
I1014 19:26:57.681118 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 429 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 429 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.681288 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 429 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 429 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 429 because replica has last-logged OpId of term: 427 index: 280, which is greater than that of the candidate, which has last-logged OpId of term: 426 index: 279.
I1014 19:26:57.681340 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 429 election: Election decided. Result: candidate lost.
I1014 19:26:57.681533 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (election complete)
I1014 19:26:57.681592 19920 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 429 FOLLOWER]: Leader election lost for term 429. Reason: could not achieve majority
I1014 19:26:57.682754 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 429 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 428. Current term is 429. Ops: []
I1014 19:26:57.683302 20178 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:57.683457 20178 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 427.280 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 279
I1014 19:26:57.683542 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Stepping down as leader of term 428
I1014 19:26:57.683589 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:57.683835 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 428.280, Last appended by leader: 280, Current term: 428, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.683992 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:57.684031 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:57.684099 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 428 FOLLOWER]: Advancing to term 429
I1014 19:26:57.693491 19667 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 429 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.693641 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:57.693701 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 429 FOLLOWER]: Advancing to term 430
I1014 19:26:57.696010 19667 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.696499 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 430 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.696849 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 430 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.697183 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 430 candidate_status { last_received { term: 427 index: 280 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.697343 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 429 FOLLOWER]: Advancing to term 430
I1014 19:26:57.697504 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 430 candidate_status { last_received { term: 427 index: 280 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.697671 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 429 FOLLOWER]: Advancing to term 430
I1014 19:26:57.697762 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:57.699321 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 430 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 430.
I1014 19:26:57.699481 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 430 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 430 because replica has last-logged OpId of term: 428 index: 280, which is greater than that of the candidate, which has last-logged OpId of term: 427 index: 280.
I1014 19:26:57.699673 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 430 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.699730 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 430 election: Election decided. Result: candidate won.
I1014 19:26:57.699903 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:26:57.699964 19667 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 FOLLOWER]: Leader election won for term 430
I1014 19:26:57.700049 19667 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:57.700260 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 430 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 430 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 430 because replica has last-logged OpId of term: 428 index: 280, which is greater than that of the candidate, which has last-logged OpId of term: 427 index: 280.
I1014 19:26:57.700282 19667 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 427.280, Last appended by leader: 280, Current term: 430, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.701068 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 430 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.701233 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.806577 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 430 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 426 index: 279. Preceding OpId from leader: term: 430 index: 281. (index mismatch)
I1014 19:26:57.806767 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 430 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.806820 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.177s (starting election)
I1014 19:26:57.806874 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 430 FOLLOWER]: Advancing to term 431
I1014 19:26:57.807183 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:57.808431 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 430 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.808495 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (starting election)
I1014 19:26:57.808542 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 430 FOLLOWER]: Advancing to term 431
I1014 19:26:57.809545 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.810034 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 431 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.810173 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 431 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.810369 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 430. Current term is 431. Ops: []
I1014 19:26:57.810612 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.810699 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 431 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.810887 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 431: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:57.811105 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 431 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.811224 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 431 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.811656 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 431 candidate_status { last_received { term: 428 index: 280 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.811785 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 431: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:57.811874 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 431 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 431: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:57.812096 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 431 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 431: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:57.812292 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 431 candidate_status { last_received { term: 428 index: 280 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.812335 17537 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 426.279, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:57.812507 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 431 candidate_status { last_received { term: 426 index: 279 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.812613 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 431 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.812671 19667 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 LEADER]: Stepping down as leader of term 430
I1014 19:26:57.812701 19667 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:57.812850 19667 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 430.281, Last appended by leader: 281, Current term: 430, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.812988 19667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:57.813024 19667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:57.813118 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 FOLLOWER]: Advancing to term 431
I1014 19:26:57.814640 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 431 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:57.814702 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 431 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 430 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 431 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.814770 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 431 election: Election decided. Result: candidate lost.
I1014 19:26:57.814950 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:57.814997 18862 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Leader election lost for term 431. Reason: could not achieve majority
I1014 19:26:57.815026 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 431 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 431 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 431 because replica is already servicing an update from a current leader or another vote.
I1014 19:26:57.815094 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 431 election: Election decided. Result: candidate lost.
I1014 19:26:57.815279 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.207s (election complete)
I1014 19:26:57.815330 19920 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Leader election lost for term 431. Reason: could not achieve majority
I1014 19:26:57.920336 18862 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.920440 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.159s (starting election)
I1014 19:26:57.920495 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 431 FOLLOWER]: Advancing to term 432
I1014 19:26:57.922704 18862 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.923214 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 432 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:57.923369 18862 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 432 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.924018 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 432 candidate_status { last_received { term: 428 index: 280 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:57.924198 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 431 FOLLOWER]: Advancing to term 432
I1014 19:26:57.924150 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 432 candidate_status { last_received { term: 428 index: 280 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.924299 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 431 FOLLOWER]: Advancing to term 432
I1014 19:26:57.924366 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.170s (vote granted)
I1014 19:26:57.926335 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 432 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 432 because replica has last-logged OpId of term: 430 index: 281, which is greater than that of the candidate, which has last-logged OpId of term: 428 index: 280.
I1014 19:26:57.926534 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 432 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 432.
I1014 19:26:57.927023 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 432 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 432 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 432 because replica has last-logged OpId of term: 430 index: 281, which is greater than that of the candidate, which has last-logged OpId of term: 428 index: 280.
I1014 19:26:57.927114 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 432 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.927170 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 432 election: Election decided. Result: candidate won.
I1014 19:26:57.927506 18862 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:57.927598 18862 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 FOLLOWER]: Leader election won for term 432
I1014 19:26:57.927809 18862 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:57.928113 18862 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 428.280, Last appended by leader: 280, Current term: 432, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.929235 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 432 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.929441 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:57.934073 19667 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 432 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:57.934144 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:57.934211 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 432 FOLLOWER]: Advancing to term 433
I1014 19:26:57.936357 19667 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.936861 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 433 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:57.937014 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 433 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.938390 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 433 candidate_status { last_received { term: 430 index: 281 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:57.938611 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 433 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.938747 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 433 candidate_status { last_received { term: 430 index: 281 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:57.938907 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 432 FOLLOWER]: Advancing to term 433
I1014 19:26:57.938982 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (vote granted)
I1014 19:26:57.938977 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 433 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 433 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:57.941344 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 433 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 433.
I1014 19:26:57.941706 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 433 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:57.941761 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 433 election: Election decided. Result: candidate won.
I1014 19:26:57.942116 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:26:57.942188 19667 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 FOLLOWER]: Leader election won for term 433
I1014 19:26:57.942275 19667 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:57.942512 19667 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 430.281, Last appended by leader: 281, Current term: 433, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:57.943564 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:57.943750 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:26:58.004246 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.908s	user 0.001s	sys 0.000s
I1014 19:26:58.004308 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:58.005198 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:26:58.010433 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 433 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 432. Current term is 433. Ops: []
I1014 19:26:58.011087 18862 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 281, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:58.011359 18862 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 LEADER]: Stepping down as leader of term 432
I1014 19:26:58.011400 18862 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:58.011598 18862 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 432.281, Last appended by leader: 281, Current term: 432, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.011791 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:58.011831 18862 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:58.011931 18862 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 432 FOLLOWER]: Advancing to term 433
I1014 19:26:58.043403 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 433 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 426 index: 279. Preceding OpId from leader: term: 433 index: 282. (index mismatch)
I1014 19:26:58.044051 17537 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:58.045666 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.045869 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.047338 17537 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.047538 17537 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.048385 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 432 index: 281. Preceding OpId from leader: term: 433 index: 282. (index mismatch)
I1014 19:26:58.048931 19667 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 279, Time since last communication: 0.000s
I1014 19:26:58.049065 19667 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 432.281 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 279
I1014 19:26:58.049988 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 279
I1014 19:26:58.050016 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 428.280
I1014 19:26:58.050053 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:58.050096 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 432.281
I1014 19:26:58.050114 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:58.069227 18862 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.069449 18862 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.069655 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.069830 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.070044 20312 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.070603 20312 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.069754 20313 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.071179 20313 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.072214 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.072425 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.072620 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.072824 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.072942 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.073108 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.073496 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 433 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.074242 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.091538 15806 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:58.093502 15806 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:58.262943 20315 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 433 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:58.263025 20315 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:26:58.263077 20315 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 433 FOLLOWER]: Advancing to term 434
I1014 19:26:58.264665 20315 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.265051 20315 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 434 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.265228 20315 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 434 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:58.265717 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 434 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:58.265740 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 434 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:58.265902 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 434 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.265908 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 434 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.266258 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 434 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 434 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.266409 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 434 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 434 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.266460 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 434 election: Election decided. Result: candidate lost.
I1014 19:26:58.266645 20315 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:58.266686 20315 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Leader election lost for term 434. Reason: could not achieve majority
I1014 19:26:58.269676 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 433. Current term is 434. Ops: []
I1014 19:26:58.270068 19667 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 433.283, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I1014 19:26:58.270248 20316 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 LEADER]: Stepping down as leader of term 433
I1014 19:26:58.270289 20316 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:58.270521 20316 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 433.283, Last appended by leader: 283, Current term: 433, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.270771 20316 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:58.270838 20316 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:58.270982 20316 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 433 FOLLOWER]: Advancing to term 434
W1014 19:26:58.279026 19783 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.659s	user 0.002s	sys 0.000s
I1014 19:26:58.279067 19783 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:58.279867 19783 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:26:58.283684 19783 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 422 vs 434: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.284036 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.284308 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.284634 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.284878 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.285116 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.285354 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.285636 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.285897 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 434 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.362447 20313 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:58.362561 20313 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (starting election)
I1014 19:26:58.362618 20313 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 433 FOLLOWER]: Advancing to term 434
I1014 19:26:58.364786 20313 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.365252 20313 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 434 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.365717 20313 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 434 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.366214 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 434 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:58.366380 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (vote granted)
I1014 19:26:58.368580 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 434 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 434.
I1014 19:26:58.369185 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 434 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:58.369454 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 434: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:58.369907 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 434 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 434: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:58.370705 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 434 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.370757 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 434 election: Election decided. Result: candidate won.
I1014 19:26:58.373152 20313 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (election complete)
I1014 19:26:58.373224 20313 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 FOLLOWER]: Leader election won for term 434
I1014 19:26:58.373317 20313 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:58.373546 20313 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 433.283, Last appended by leader: 283, Current term: 434, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.374774 20312 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 434 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.374956 20312 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.392053 20315 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:58.392144 20315 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:26:58.392196 20315 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 434 FOLLOWER]: Advancing to term 435
I1014 19:26:58.394168 20315 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.394590 20315 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 435 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.394721 20315 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 435 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:58.395392 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 435 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:58.395570 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 434 FOLLOWER]: Advancing to term 435
I1014 19:26:58.395639 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:26:58.395705 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 435 candidate_status { last_received { term: 433 index: 283 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:58.395900 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 435 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.396438 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 435 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 435 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.397434 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 435.
I1014 19:26:58.397761 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 435 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.397809 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 435 election: Election decided. Result: candidate won.
I1014 19:26:58.399977 20315 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (election complete)
I1014 19:26:58.400044 20315 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 FOLLOWER]: Leader election won for term 435
I1014 19:26:58.400292 20315 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:58.400540 20315 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 433.283, Last appended by leader: 283, Current term: 435, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.405519 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.405675 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.406188 19783 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:58.407809 19783 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:26:58.434185 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.829s	user 0.001s	sys 0.001s
I1014 19:26:58.434232 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:58.435112 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:58.438102 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 434. Current term is 435. Ops: []
I1014 19:26:58.438102 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 434. Current term is 435. Ops: []
I1014 19:26:58.438717 20313 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I1014 19:26:58.438884 20313 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 435.284 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 283
I1014 19:26:58.438948 20312 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 LEADER]: Stepping down as leader of term 434
I1014 19:26:58.439020 20312 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:58.439209 20312 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 434.285, Last appended by leader: 285, Current term: 434, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.439422 20312 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:58.439471 20313 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I1014 19:26:58.439848 20312 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:58.439980 20312 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 434 FOLLOWER]: Advancing to term 435
I1014 19:26:58.491302 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 435 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 434 index: 285. Preceding OpId from leader: term: 435 index: 284. (term mismatch)
I1014 19:26:58.491351 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 283
I1014 19:26:58.491370 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 434.284
I1014 19:26:58.491407 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 435 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:58.491472 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 434.285
I1014 19:26:58.491806 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 421 vs 435: Aborted: Transaction aborted by new leader
W1014 19:26:58.492075 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 435 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:26:58.492267 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 435 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.492265 20315 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
W1014 19:26:58.492404 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 435 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.493603 20312 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.493757 20312 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.494603 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.494935 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.495283 20312 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.495430 20312 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.495534 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 433 index: 283. Preceding OpId from leader: term: 435 index: 284. (index mismatch)
I1014 19:26:58.495965 20315 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I1014 19:26:58.497180 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.497309 19667 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.497503 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.497519 19667 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.586676 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 435 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.586915 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.595420 20312 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 435 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:58.595526 20312 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:26:58.595579 20312 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 435 FOLLOWER]: Advancing to term 436
I1014 19:26:58.597682 20312 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.598172 20312 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 436 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.598338 20312 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 436 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.598867 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 436 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:58.598870 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 436 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:58.599058 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 436 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.599090 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 436 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.599373 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 436 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 436 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.599535 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 436 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 436 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.599601 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 436 election: Election decided. Result: candidate lost.
I1014 19:26:58.599762 20312 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:58.599823 20312 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Leader election lost for term 436. Reason: could not achieve majority
I1014 19:26:58.602555 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 435. Current term is 436. Ops: []
I1014 19:26:58.603363 20314 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 435.284, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I1014 19:26:58.603632 20314 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Stepping down as leader of term 435
I1014 19:26:58.603675 20314 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:58.603847 20314 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 435.284, Last appended by leader: 284, Current term: 435, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.604043 20314 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:58.604085 20314 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:58.604213 20314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 435 FOLLOWER]: Advancing to term 436
I1014 19:26:58.688047 19667 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:58.688128 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:26:58.688176 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 435 FOLLOWER]: Advancing to term 436
I1014 19:26:58.690446 19667 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.690951 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 436 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:58.691115 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 436 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.691722 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 436 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:58.691722 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 436 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:58.691915 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 436: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:58.691889 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (vote granted)
I1014 19:26:58.692333 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 436 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 436: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:58.693516 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 436 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 436.
I1014 19:26:58.693809 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 436 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.693861 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 436 election: Election decided. Result: candidate won.
I1014 19:26:58.694021 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (election complete)
I1014 19:26:58.694077 19667 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 FOLLOWER]: Leader election won for term 436
I1014 19:26:58.694186 19667 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:58.694835 19667 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 435.284, Last appended by leader: 284, Current term: 436, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.696468 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 436 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.696661 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.746064 20312 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:58.746140 20312 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:26:58.746192 20312 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 436 FOLLOWER]: Advancing to term 437
I1014 19:26:58.747889 20312 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.748289 20312 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 437 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:58.748438 20312 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 437 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.748946 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 437 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:58.749148 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 437 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.749464 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 437 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 437 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.749997 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 437 candidate_status { last_received { term: 435 index: 284 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:58.750136 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 436 FOLLOWER]: Advancing to term 437
I1014 19:26:58.750211 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (vote granted)
I1014 19:26:58.751827 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 437 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 437.
I1014 19:26:58.752142 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 437 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.752192 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 437 election: Election decided. Result: candidate won.
I1014 19:26:58.752348 20312 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:26:58.752403 20312 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 FOLLOWER]: Leader election won for term 437
I1014 19:26:58.752504 20312 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:58.752668 20312 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 435.284, Last appended by leader: 284, Current term: 437, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.753510 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 437 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.753688 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.753846 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:26:58.755198 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:26:58.774420 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 436. Current term is 437. Ops: []
I1014 19:26:58.774938 20316 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I1014 19:26:58.775121 20316 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 437.285 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 284
I1014 19:26:58.775173 19667 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 LEADER]: Stepping down as leader of term 436
I1014 19:26:58.775215 19667 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:58.775387 19667 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 436.285, Last appended by leader: 285, Current term: 436, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.775652 19667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:58.775692 19667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:58.775769 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 436 FOLLOWER]: Advancing to term 437
I1014 19:26:58.851361 19667 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 437 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:58.851445 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:26:58.851495 19667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 437 FOLLOWER]: Advancing to term 438
I1014 19:26:58.853052 19667 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.853474 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 438 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:58.853631 19667 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 438 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.854153 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 438 candidate_status { last_received { term: 436 index: 285 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:58.854220 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 438 candidate_status { last_received { term: 436 index: 285 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:58.854362 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 438 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.854369 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 437 FOLLOWER]: Advancing to term 438
I1014 19:26:58.854431 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (vote granted)
I1014 19:26:58.854826 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 438 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 438 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:58.856600 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 438 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 438.
I1014 19:26:58.856910 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 438 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:58.856964 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 438 election: Election decided. Result: candidate won.
I1014 19:26:58.857137 19667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:26:58.857187 19667 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 FOLLOWER]: Leader election won for term 438
I1014 19:26:58.857255 19667 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:58.857456 19667 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 436.285, Last appended by leader: 285, Current term: 438, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.858319 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.858532 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.859742 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 438 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 437. Current term is 438. Ops: []
I1014 19:26:58.860188 20178 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I1014 19:26:58.860376 20178 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Stepping down as leader of term 437
I1014 19:26:58.860414 20178 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:58.860592 20178 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 437.285, Last appended by leader: 285, Current term: 437, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:58.860760 20178 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:58.860795 20178 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:58.860888 20178 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 437 FOLLOWER]: Advancing to term 438
I1014 19:26:58.952080 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 438 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 435 index: 284. Preceding OpId from leader: term: 438 index: 286. (index mismatch)
I1014 19:26:58.952527 20316 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 284, Time since last communication: 0.000s
I1014 19:26:58.953661 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.953997 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.954840 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.955051 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:58.955325 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 437 index: 285. Preceding OpId from leader: term: 438 index: 286. (index mismatch)
I1014 19:26:58.955453 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.955664 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.955839 20316 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 284, Time since last communication: 0.000s
I1014 19:26:58.955941 20316 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 437.285 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 284
I1014 19:26:58.955824 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.955978 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.956531 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 284
I1014 19:26:58.956562 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 437.285
I1014 19:26:58.956598 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:58.957686 20312 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.957682 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.957867 20312 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.957893 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:58.957950 20466 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:58.958129 20466 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.034075 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 438 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.034334 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.055799 20314 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 438 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:59.055912 20314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:26:59.055977 20314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 438 FOLLOWER]: Advancing to term 439
I1014 19:26:59.058303 20314 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.058853 20314 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 439 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.059231 20314 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 439 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:59.059572 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 439 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.059752 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 439 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.059809 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 439 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:59.059979 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 439 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.060137 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 439 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 439 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.060297 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 439 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 439 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.060338 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 439 election: Election decided. Result: candidate lost.
I1014 19:26:59.060506 20314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:26:59.060559 20314 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Leader election lost for term 439. Reason: could not achieve majority
I1014 19:26:59.063022 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 438. Current term is 439. Ops: []
I1014 19:26:59.063433 20316 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 438.286, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I1014 19:26:59.063678 20469 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 LEADER]: Stepping down as leader of term 438
I1014 19:26:59.063717 20469 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:59.063889 20469 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 438.286, Last appended by leader: 286, Current term: 438, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.064028 20469 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:59.064085 20469 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:59.064189 20469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 438 FOLLOWER]: Advancing to term 439
I1014 19:26:59.137671 20466 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:59.137759 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:26:59.137820 20466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 438 FOLLOWER]: Advancing to term 439
I1014 19:26:59.140044 20466 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.140583 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 439 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.140743 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 439 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.141882 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 439 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:59.142091 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 439: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:59.142298 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 439 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.142496 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (vote granted)
I1014 19:26:59.143224 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 439 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 439: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:26:59.144496 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 439 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 439.
I1014 19:26:59.144786 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 439 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.144837 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 439 election: Election decided. Result: candidate won.
I1014 19:26:59.145020 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (election complete)
I1014 19:26:59.145081 20466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 FOLLOWER]: Leader election won for term 439
I1014 19:26:59.145196 20466 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:59.145385 20466 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 438.286, Last appended by leader: 286, Current term: 439, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.146201 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 439 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.146369 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.165436 20314 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:59.165567 20314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.190s (starting election)
I1014 19:26:59.165648 20314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 439 FOLLOWER]: Advancing to term 440
I1014 19:26:59.167784 20314 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.168256 20314 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 440 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.168412 20314 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 440 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:59.169116 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 440 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.169142 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 440 candidate_status { last_received { term: 438 index: 286 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:59.169251 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 439 FOLLOWER]: Advancing to term 440
I1014 19:26:59.169317 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 440 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.169334 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (vote granted)
I1014 19:26:59.169745 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 440 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 440 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.170976 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 440 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 440.
I1014 19:26:59.171259 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 440 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.171310 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 440 election: Election decided. Result: candidate won.
I1014 19:26:59.171551 20314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.197s (election complete)
I1014 19:26:59.171617 20314 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 FOLLOWER]: Leader election won for term 440
I1014 19:26:59.171690 20314 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:59.171835 20314 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 438.286, Last appended by leader: 286, Current term: 440, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.172693 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.172876 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.239888 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 440 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 439. Current term is 440. Ops: []
I1014 19:26:59.240533 20178 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I1014 19:26:59.240804 20178 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 LEADER]: Stepping down as leader of term 439
I1014 19:26:59.240845 20178 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:59.241020 20178 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 439.287, Last appended by leader: 287, Current term: 439, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.241149 20178 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:59.241179 20178 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:59.241277 20178 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 439 FOLLOWER]: Advancing to term 440
I1014 19:26:59.249845 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 439 index: 287. Preceding OpId from leader: term: 440 index: 287. (term mismatch)
I1014 19:26:59.249892 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 286
I1014 19:26:59.249909 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 439.287
I1014 19:26:59.249943 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:59.250491 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I1014 19:26:59.251637 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.251837 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.252784 20314 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.252977 20314 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.253173 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.253334 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.253772 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 440 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 438 index: 286. Preceding OpId from leader: term: 440 index: 287. (index mismatch)
I1014 19:26:59.254287 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I1014 19:26:59.255589 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.255781 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.255856 20469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.256033 20469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.352474 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 440 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.352741 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.454393 20469 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 440 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:59.454500 20469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:26:59.454560 20469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 440 FOLLOWER]: Advancing to term 441
I1014 19:26:59.456221 20469 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.456749 20469 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 441 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:59.456897 20469 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 441 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.457621 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 440. Current term is 441. Ops: []
I1014 19:26:59.457584 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 441 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:59.457587 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 441 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:59.457754 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 441 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.457777 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 441 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.458083 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 441 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 441 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.458186 19920 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 440.287, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I1014 19:26:59.458344 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 441 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 441 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.458402 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 441 election: Election decided. Result: candidate lost.
I1014 19:26:59.458433 19920 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 LEADER]: Stepping down as leader of term 440
I1014 19:26:59.458468 19920 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:59.458585 20469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (election complete)
I1014 19:26:59.458636 20469 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Leader election lost for term 441. Reason: could not achieve majority
I1014 19:26:59.458657 19920 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 440.287, Last appended by leader: 287, Current term: 440, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.458827 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:59.458866 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:59.458979 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 440 FOLLOWER]: Advancing to term 441
I1014 19:26:59.532193 20178 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:26:59.532279 20178 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:26:59.532333 20178 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 440 FOLLOWER]: Advancing to term 441
I1014 19:26:59.533803 20178 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.534164 20178 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 441 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.534287 20178 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 441 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.535094 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 441 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.535135 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 441 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:59.535293 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 441: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:59.535293 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (vote granted)
I1014 19:26:59.535791 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 441 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 441: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:26:59.537334 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 441 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 441.
I1014 19:26:59.537725 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 441 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.537775 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 441 election: Election decided. Result: candidate won.
I1014 19:26:59.537976 20178 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:26:59.538033 20178 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 FOLLOWER]: Leader election won for term 441
I1014 19:26:59.538127 20178 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:59.538313 20178 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 440.287, Last appended by leader: 287, Current term: 441, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.539124 20466 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 441 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.539278 20466 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.591222 20469 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:26:59.591341 20469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.158s (starting election)
I1014 19:26:59.591403 20469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 441 FOLLOWER]: Advancing to term 442
I1014 19:26:59.592984 20469 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.593466 20469 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 442 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:59.593621 20469 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 442 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.594188 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 442 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:59.594177 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 442 candidate_status { last_received { term: 440 index: 287 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:59.594324 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 441 FOLLOWER]: Advancing to term 442
I1014 19:26:59.594355 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 442 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.594396 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (vote granted)
I1014 19:26:59.594710 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 442 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 442 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.595984 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 442.
I1014 19:26:59.596263 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 442 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.596315 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 442 election: Election decided. Result: candidate won.
I1014 19:26:59.596602 20469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:26:59.596665 20469 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 FOLLOWER]: Leader election won for term 442
I1014 19:26:59.596745 20469 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:59.596930 20469 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 440.287, Last appended by leader: 287, Current term: 442, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.598109 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.598285 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.605952 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:26:59.606772 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:26:59.609206 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 441. Current term is 442. Ops: []
I1014 19:26:59.609273 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 441. Current term is 442. Ops: []
I1014 19:26:59.609772 20466 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I1014 19:26:59.610167 20178 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I1014 19:26:59.610316 20178 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 442.288 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 287
I1014 19:26:59.610400 20466 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 LEADER]: Stepping down as leader of term 441
I1014 19:26:59.610436 20466 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:26:59.610616 20466 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 441.289, Last appended by leader: 289, Current term: 441, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.610743 20466 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:59.610777 20466 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:26:59.610872 20466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 441 FOLLOWER]: Advancing to term 442
I1014 19:26:59.678230 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 442 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 441 index: 289. Preceding OpId from leader: term: 442 index: 288. (term mismatch)
I1014 19:26:59.678298 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 287
I1014 19:26:59.678320 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 441.288
I1014 19:26:59.678361 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 442 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:26:59.678406 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 441.289
I1014 19:26:59.678784 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 437 vs 442: Aborted: Transaction aborted by new leader
W1014 19:26:59.679067 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 442 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.679198 20316 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
W1014 19:26:59.679293 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 442 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.680258 20466 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.680511 20466 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.681427 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.681638 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.681726 20466 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.681895 20466 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.682313 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 440 index: 287. Preceding OpId from leader: term: 442 index: 288. (index mismatch)
I1014 19:26:59.682711 20469 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I1014 19:26:59.684058 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.684254 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.684356 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.684581 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:26:59.768136 20469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 442 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.768338 20469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:26:59.865833 20466 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 442 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:59.865945 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:26:59.865993 20466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 442 FOLLOWER]: Advancing to term 443
I1014 19:26:59.868193 20466 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.868705 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 443 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.868861 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 443 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:26:59.869673 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 443 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.869858 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 443 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.869995 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 442. Current term is 443. Ops: []
I1014 19:26:59.870251 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 443 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:26:59.870482 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 443 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.870575 20496 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 442.288, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I1014 19:26:59.870784 20316 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Stepping down as leader of term 442
I1014 19:26:59.870823 20316 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:26:59.870985 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 443 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 443 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.870990 20316 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 442.288, Last appended by leader: 288, Current term: 442, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.871162 20316 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:26:59.871198 20316 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:26:59.871305 20316 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 FOLLOWER]: Advancing to term 443
I1014 19:26:59.873769 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 443 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 442 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 443 because replica is either leader or believes a valid leader to be alive.
I1014 19:26:59.873850 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 443 election: Election decided. Result: candidate lost.
I1014 19:26:59.874048 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:26:59.874131 20466 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Leader election lost for term 443. Reason: could not achieve majority
I1014 19:26:59.957947 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:26:59.958042 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:26:59.958093 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 442 FOLLOWER]: Advancing to term 443
I1014 19:26:59.960366 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.960889 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 443 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.961056 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 443 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:26:59.961670 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 443 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:26:59.961670 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 443 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:26:59.961818 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (vote granted)
I1014 19:26:59.961848 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 443: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:59.962283 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 443 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 443: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:26:59.963346 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 443 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 443.
I1014 19:26:59.963645 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 443 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:26:59.963698 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 443 election: Election decided. Result: candidate won.
I1014 19:26:59.963871 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (election complete)
I1014 19:26:59.963929 19920 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 FOLLOWER]: Leader election won for term 443
I1014 19:26:59.964020 19920 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:26:59.964205 19920 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 442.288, Last appended by leader: 288, Current term: 443, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:26:59.965204 20480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 443 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:26:59.965397 20480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.007676 20466 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:00.007761 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (starting election)
I1014 19:27:00.007808 20466 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 443 FOLLOWER]: Advancing to term 444
I1014 19:27:00.009709 20466 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.010210 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 444 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.010377 20466 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 444 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.010936 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 444 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:00.010983 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 444 candidate_status { last_received { term: 442 index: 288 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.011072 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 443 FOLLOWER]: Advancing to term 444
I1014 19:27:00.011138 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (vote granted)
I1014 19:27:00.011173 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 444 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.011595 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 444 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 444 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.012624 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 444 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 444.
I1014 19:27:00.012938 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 444 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.012995 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 444 election: Election decided. Result: candidate won.
I1014 19:27:00.013164 20466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:00.013218 20466 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 FOLLOWER]: Leader election won for term 444
I1014 19:27:00.013305 20466 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:00.013514 20466 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 442.288, Last appended by leader: 288, Current term: 444, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.014421 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.014571 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.014737 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:00.016544 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:00.043107 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 443. Current term is 444. Ops: []
I1014 19:27:00.043740 20480 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I1014 19:27:00.043936 20480 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 444.289 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 288
I1014 19:27:00.044019 20480 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 LEADER]: Stepping down as leader of term 443
I1014 19:27:00.044052 20480 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:00.044235 20480 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 443.289, Last appended by leader: 289, Current term: 443, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.044440 20480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:00.044494 20480 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:00.044598 20480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 443 FOLLOWER]: Advancing to term 444
W1014 19:27:00.090284 15806 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.997s	user 0.001s	sys 0.000s
I1014 19:27:00.090337 15806 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:00.091146 15806 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:00.092200 15806 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 433 vs 444: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.092445 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.092656 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.092835 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.093024 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.093219 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.093408 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.093621 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.093811 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.094002 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.094183 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.094364 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.094581 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.094774 15806 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 444 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.101994 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 443 index: 289. Preceding OpId from leader: term: 444 index: 289. (term mismatch)
I1014 19:27:00.102039 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 288
I1014 19:27:00.102053 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 443.289
I1014 19:27:00.102082 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:00.102622 20178 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I1014 19:27:00.103713 20480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.103956 20480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.104794 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.104985 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.105379 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 444 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 442 index: 288. Preceding OpId from leader: term: 444 index: 289. (index mismatch)
I1014 19:27:00.105715 20480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.105880 20480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.106063 20490 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s
I1014 19:27:00.107455 20316 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.107657 20316 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.107509 20496 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.107761 20496 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.207715 20496 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 444 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:00.207808 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:00.207871 20496 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 444 FOLLOWER]: Advancing to term 445
I1014 19:27:00.209522 20496 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.209956 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 445 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.210139 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 445 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.210837 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 445 candidate_status { last_received { term: 444 index: 289 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.210839 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 445 candidate_status { last_received { term: 444 index: 289 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:00.211006 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 445 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.211009 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 445 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.211382 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 445 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 445 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.211954 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 445 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 445 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.212028 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 445 election: Election decided. Result: candidate lost.
I1014 19:27:00.212272 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:00.212335 20496 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Leader election lost for term 445. Reason: could not achieve majority
I1014 19:27:00.224745 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 444 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.224973 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.225323 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 444. Current term is 445. Ops: []
I1014 19:27:00.225816 20178 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 444.289, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I1014 19:27:00.226003 20490 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Stepping down as leader of term 444
I1014 19:27:00.226047 20490 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:00.226224 20490 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 444.289, Last appended by leader: 289, Current term: 444, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.226436 20490 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:00.226496 20490 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:00.226619 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 444 FOLLOWER]: Advancing to term 445
I1014 19:27:00.282145 20480 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:00.282228 20480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:00.282274 20480 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 444 FOLLOWER]: Advancing to term 445
I1014 19:27:00.284657 20480 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.285185 20480 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 445 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.285379 20480 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 445 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.285950 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 445 candidate_status { last_received { term: 444 index: 289 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:00.286130 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 445: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:00.286535 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 445 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 445: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:00.287256 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 445 candidate_status { last_received { term: 444 index: 289 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:00.287395 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (vote granted)
I1014 19:27:00.289073 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 445.
I1014 19:27:00.289387 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 445 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.289458 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 445 election: Election decided. Result: candidate won.
I1014 19:27:00.289635 20480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:00.289695 20480 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 FOLLOWER]: Leader election won for term 445
I1014 19:27:00.289777 20480 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:00.289943 20480 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 444.289, Last appended by leader: 289, Current term: 445, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.290879 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.291095 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:27:00.304633 19783 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.897s	user 0.002s	sys 0.000s
I1014 19:27:00.304682 19783 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:00.305577 19783 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:00.308008 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 444 index: 289. Preceding OpId from leader: term: 445 index: 291. (index mismatch)
I1014 19:27:00.308008 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 444 index: 289. Preceding OpId from leader: term: 445 index: 291. (index mismatch)
I1014 19:27:00.308626 20480 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I1014 19:27:00.308836 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s
I1014 19:27:00.310706 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.310901 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.311072 20496 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.311270 20496 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.321851 19783 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:00.321951 20480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.322171 20480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.322312 20480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.322463 20480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.323846 19783 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:00.325847 20496 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.326066 20496 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.329335 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 445 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.329536 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.507931 20496 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:00.508016 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:00.508071 20496 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 445 FOLLOWER]: Advancing to term 446
I1014 19:27:00.510193 20496 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.510741 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 446 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.510890 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 446 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.511584 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 446 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.511584 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 446 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:00.511770 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 446 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.511770 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 446 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.512181 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 446 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 446 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.512256 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 446 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 446 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.512312 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 446 election: Election decided. Result: candidate lost.
I1014 19:27:00.512650 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:00.512732 20496 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Leader election lost for term 446. Reason: could not achieve majority
I1014 19:27:00.516196 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 445. Current term is 446. Ops: []
I1014 19:27:00.516649 19920 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 445.291, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I1014 19:27:00.516882 20583 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 LEADER]: Stepping down as leader of term 445
I1014 19:27:00.516923 20583 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:00.517097 20583 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 445.291, Last appended by leader: 291, Current term: 445, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.517271 20583 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:00.517307 20583 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:00.517443 20583 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 445 FOLLOWER]: Advancing to term 446
I1014 19:27:00.540803 19783 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:00.541784 19783 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:00.542872 19783 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 445 vs 446: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.543130 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.543324 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.543550 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.543730 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.543917 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.544101 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.544279 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.544734 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 446 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.608467 20490 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:00.608587 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:00.608649 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 445 FOLLOWER]: Advancing to term 446
I1014 19:27:00.610844 20490 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.611343 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 446 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.611733 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 446 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.612054 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 446 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:00.612272 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 446: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:00.612283 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 446 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.612433 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (vote granted)
I1014 19:27:00.612614 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 446 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 446: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:00.613924 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 446 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 446.
I1014 19:27:00.614202 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 446 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.614249 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 446 election: Election decided. Result: candidate won.
I1014 19:27:00.614418 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:00.614471 20490 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 FOLLOWER]: Leader election won for term 446
I1014 19:27:00.614578 20490 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:00.614727 20490 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 445.291, Last appended by leader: 291, Current term: 446, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.615444 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 446 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.615610 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.662305 20496 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:00.662392 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.174s (starting election)
I1014 19:27:00.662451 20496 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 446 FOLLOWER]: Advancing to term 447
I1014 19:27:00.664441 20496 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.664974 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 447 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.665136 20496 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 447 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.665751 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 447 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.665753 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 447 candidate_status { last_received { term: 445 index: 291 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:00.665892 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 446 FOLLOWER]: Advancing to term 447
I1014 19:27:00.665920 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 447 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.665964 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (vote granted)
I1014 19:27:00.666301 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 447 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 447 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.667850 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 447.
I1014 19:27:00.668135 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 447 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.668186 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 447 election: Election decided. Result: candidate won.
I1014 19:27:00.668365 20496 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:00.668423 20496 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 FOLLOWER]: Leader election won for term 447
I1014 19:27:00.668817 20496 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:00.669044 20496 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 445.291, Last appended by leader: 291, Current term: 447, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.669992 20605 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.670187 20605 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.670709 20606 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:00.672029 20606 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:00.679703 20606 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:00.681344 20606 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:00.683341 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 445 index: 291. Preceding OpId from leader: term: 447 index: 293. (index mismatch)
I1014 19:27:00.683451 28825 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 LEADER]: Stepping down as leader of term 446
I1014 19:27:00.683508 28825 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:00.683678 28825 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 446.292, Last appended by leader: 292, Current term: 446, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.683822 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:00.683871 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:00.683845 20605 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I1014 19:27:00.683943 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 446 FOLLOWER]: Advancing to term 447
I1014 19:27:00.685199 20583 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.685395 20583 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.686077 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 447 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 446 index: 292. Preceding OpId from leader: term: 447 index: 293. (index mismatch)
I1014 19:27:00.687103 20583 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.687288 20583 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.687595 20496 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.687775 20496 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.687737 20605 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I1014 19:27:00.687845 20605 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 446.292 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 291
I1014 19:27:00.688943 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 291
I1014 19:27:00.688977 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 446.292
I1014 19:27:00.689016 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 447 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:00.689945 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.690083 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.690178 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.690271 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:00.788870 20608 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 447 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.789125 20608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.891301 20490 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 447 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:00.891386 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:00.891443 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 447 FOLLOWER]: Advancing to term 448
I1014 19:27:00.893651 20490 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.894223 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 448 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.894374 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 448 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:00.894955 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 448 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:00.894955 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 448 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:00.895140 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 448 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.895145 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 448 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.895587 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 448 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 448 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.895691 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 448 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 448 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:00.895768 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 448 election: Election decided. Result: candidate lost.
I1014 19:27:00.895959 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:00.896013 20490 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Leader election lost for term 448. Reason: could not achieve majority
I1014 19:27:00.900274 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:00.901170 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:00.902142 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 444 vs 448: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 448 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.902359 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 448 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.903183 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 448 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:00.903468 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 448 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.903798 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 447. Current term is 448. Ops: []
I1014 19:27:00.904317 20608 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 447.293, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I1014 19:27:00.904575 20607 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 LEADER]: Stepping down as leader of term 447
I1014 19:27:00.904621 20607 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:00.904822 20607 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 447.293, Last appended by leader: 293, Current term: 447, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.905014 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:00.905055 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:00.905181 20607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 447 FOLLOWER]: Advancing to term 448
I1014 19:27:00.960610 20583 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:00.960688 20583 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:00.960729 20583 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 447 FOLLOWER]: Advancing to term 448
I1014 19:27:00.962249 20583 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.962682 20583 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 448 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.962838 20583 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 448 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:00.963418 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 448 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:00.963488 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 448 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:00.963591 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (vote granted)
I1014 19:27:00.963675 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 448: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:00.965775 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 448 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 448.
I1014 19:27:00.966189 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 448 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:00.966253 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 448 election: Election decided. Result: candidate won.
I1014 19:27:00.966502 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 448 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 448: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:00.966543 20583 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:00.966599 20583 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 FOLLOWER]: Leader election won for term 448
I1014 19:27:00.966691 20583 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:00.966881 20583 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 447.293, Last appended by leader: 293, Current term: 448, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:00.968796 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 448 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:00.968988 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:00.969197 19783 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:00.971115 19783 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:01.000223 20490 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:01.000317 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:01.000380 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 448 FOLLOWER]: Advancing to term 449
I1014 19:27:01.002528 20490 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.003021 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 449 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.003185 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 449 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.003840 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 449 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.003840 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 449 candidate_status { last_received { term: 447 index: 293 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.003995 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 448 FOLLOWER]: Advancing to term 449
I1014 19:27:01.004036 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 449 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.004062 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:01.004557 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 449 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 449 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.005673 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 449 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 449.
I1014 19:27:01.005961 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 449 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.006014 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 449 election: Election decided. Result: candidate won.
I1014 19:27:01.006193 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.209s (election complete)
I1014 19:27:01.006253 20490 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 FOLLOWER]: Leader election won for term 449
I1014 19:27:01.006342 20490 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:01.006566 20490 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 447.293, Last appended by leader: 293, Current term: 449, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.007558 20178 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.007740 20178 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.007912 15726 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:01.010018 15726 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:01.067165 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 449 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 448. Current term is 449. Ops: []
I1014 19:27:01.068447 19920 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I1014 19:27:01.068722 19920 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 LEADER]: Stepping down as leader of term 448
I1014 19:27:01.068825 19920 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:01.069020 19920 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 448.294, Last appended by leader: 294, Current term: 448, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.069211 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:01.069252 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:01.069365 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 448 FOLLOWER]: Advancing to term 449
I1014 19:27:01.101219 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 448 index: 294. Preceding OpId from leader: term: 449 index: 294. (term mismatch)
I1014 19:27:01.101275 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 293
I1014 19:27:01.101296 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 448.294
I1014 19:27:01.101332 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:01.102048 20178 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I1014 19:27:01.103314 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.103653 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.104105 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 449 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 447 index: 293. Preceding OpId from leader: term: 449 index: 294. (index mismatch)
I1014 19:27:01.104902 20490 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I1014 19:27:01.105621 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.105808 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.106397 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.106657 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.106761 20608 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.106933 20608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.107784 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.107976 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.206609 20608 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 449 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:01.206692 20608 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:01.206743 20608 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 449 FOLLOWER]: Advancing to term 450
I1014 19:27:01.208988 20608 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.209470 20608 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 450 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.209854 20608 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 450 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.209990 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 449 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.210196 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.210147 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 450 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.210302 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 450 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.210608 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 449. Current term is 450. Ops: []
I1014 19:27:01.210722 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 450 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 450 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.210830 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 450 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.211021 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 450 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.211269 20490 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 449.294, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I1014 19:27:01.211338 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 450 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 450 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.211396 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 450 election: Election decided. Result: candidate lost.
I1014 19:27:01.211469 20660 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 LEADER]: Stepping down as leader of term 449
I1014 19:27:01.211509 20660 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:01.211694 20608 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:01.211742 20608 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Leader election lost for term 450. Reason: could not achieve majority
I1014 19:27:01.211660 20660 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 449.294, Last appended by leader: 294, Current term: 449, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.211866 20660 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:01.211897 20660 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:01.211997 20660 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 449 FOLLOWER]: Advancing to term 450
I1014 19:27:01.295142 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:01.295233 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:01.295289 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 449 FOLLOWER]: Advancing to term 450
I1014 19:27:01.296979 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.297541 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 450 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.297709 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 450 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.298403 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 450 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.298611 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 450: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:01.298646 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 450 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.298804 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:01.298995 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 450 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 450: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:01.300943 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 450 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 450.
I1014 19:27:01.301357 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 450 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.301420 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 450 election: Election decided. Result: candidate won.
I1014 19:27:01.301659 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:01.301718 19920 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 FOLLOWER]: Leader election won for term 450
I1014 19:27:01.301823 19920 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:01.302006 19920 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 449.294, Last appended by leader: 294, Current term: 450, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.302999 20583 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 450 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.303280 20583 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.325423 20608 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:01.325531 20608 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:01.325598 20608 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 450 FOLLOWER]: Advancing to term 451
I1014 19:27:01.327777 20608 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.328313 20608 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 451 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.328485 20608 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 451 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.328902 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 451 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.329100 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 450 FOLLOWER]: Advancing to term 451
I1014 19:27:01.329164 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (vote granted)
I1014 19:27:01.329125 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 451 candidate_status { last_received { term: 449 index: 294 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.329304 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 451 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.329798 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 451 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 451 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.331228 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 451.
I1014 19:27:01.331542 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 451 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.331599 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 451 election: Election decided. Result: candidate won.
I1014 19:27:01.331805 20608 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (election complete)
I1014 19:27:01.331871 20608 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 FOLLOWER]: Leader election won for term 451
I1014 19:27:01.331959 20608 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:01.332147 20608 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 449.294, Last appended by leader: 294, Current term: 451, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.333149 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.333322 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.335115 20771 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:01.336928 20771 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:01.381419 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 450. Current term is 451. Ops: []
I1014 19:27:01.381999 20583 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I1014 19:27:01.382215 20583 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 451.295 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 294
I1014 19:27:01.382282 19920 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 LEADER]: Stepping down as leader of term 450
I1014 19:27:01.382338 19920 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:01.382620 19920 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 450.295, Last appended by leader: 295, Current term: 450, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.382854 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:01.382930 19920 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:01.383077 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 450 FOLLOWER]: Advancing to term 451
I1014 19:27:01.429705 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 451 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 450 index: 295. Preceding OpId from leader: term: 451 index: 295. (term mismatch)
I1014 19:27:01.429772 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 294
I1014 19:27:01.429795 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 450.295
I1014 19:27:01.429836 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 451 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:01.430452 20607 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I1014 19:27:01.431952 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.432287 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.433241 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.433857 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.434245 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 449 index: 294. Preceding OpId from leader: term: 451 index: 295. (index mismatch)
I1014 19:27:01.434854 20607 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I1014 19:27:01.436421 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.436394 20660 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.436645 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.436658 20660 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.437283 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.437530 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.532444 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 451 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.532781 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.534173 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 451 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:01.534250 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:01.534307 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 451 FOLLOWER]: Advancing to term 452
I1014 19:27:01.535900 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 452 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.536367 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 452 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.536551 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 452 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.537173 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 452 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.537359 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.537324 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 452 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.537519 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.538030 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 452 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.538202 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 452 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.538266 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 452 election: Election decided. Result: candidate lost.
I1014 19:27:01.538478 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:01.538537 19920 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 452 FOLLOWER]: Leader election lost for term 452. Reason: could not achieve majority
I1014 19:27:01.551982 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 452 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 451. Current term is 452. Ops: []
I1014 19:27:01.552451 20607 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 451.295, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I1014 19:27:01.552711 20607 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Stepping down as leader of term 451
I1014 19:27:01.552749 20607 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:01.552914 20607 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 451.295, Last appended by leader: 295, Current term: 451, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.553078 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:01.553108 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:01.553207 20607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 451 FOLLOWER]: Advancing to term 452
I1014 19:27:01.631454 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 452 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:01.631537 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:01.631589 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 452 FOLLOWER]: Advancing to term 453
I1014 19:27:01.633756 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.634301 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 453 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.634480 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 453 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.634924 20660 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:01.634996 20660 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:01.635051 20660 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 451 FOLLOWER]: Advancing to term 452
I1014 19:27:01.635612 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 453 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.637212 20660 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 452 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.637653 20660 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 452 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.637806 20660 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 452 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.638085 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 452 FOLLOWER]: Advancing to term 453
I1014 19:27:01.638151 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (vote granted)
I1014 19:27:01.638519 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 453 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.638656 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 452 FOLLOWER]: Advancing to term 453
I1014 19:27:01.638711 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (vote granted)
I1014 19:27:01.640149 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 453 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 453.
I1014 19:27:01.640584 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 453 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.640640 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 453 election: Election decided. Result: candidate won.
I1014 19:27:01.640831 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 453.
I1014 19:27:01.641032 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 452 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.641167 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:01.641198 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 452. Current term is 453.
I1014 19:27:01.641207 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 453 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.641227 20772 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 FOLLOWER]: Leader election won for term 453
I1014 19:27:01.641396 20772 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
W1014 19:27:01.641651 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 452 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 452. Current term is 453.
I1014 19:27:01.641687 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 452 election: Cancelling election due to peer responding with higher term
I1014 19:27:01.641633 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 452 candidate_status { last_received { term: 451 index: 295 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.641857 20660 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:01.641636 20772 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 451.295, Last appended by leader: 295, Current term: 453, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.641917 20660 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 453 FOLLOWER]: Leader election lost for term 452. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 452. Current term is 453.
I1014 19:27:01.642762 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.642726 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.642897 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
W1014 19:27:01.643154 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 452 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 452 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.720791 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 451 index: 295. Preceding OpId from leader: term: 453 index: 296. (index mismatch)
I1014 19:27:01.721307 20772 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I1014 19:27:01.722654 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.722844 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.723749 20608 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.723945 20608 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.724500 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.724702 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.724851 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 453 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 451 index: 295. Preceding OpId from leader: term: 453 index: 296. (index mismatch)
I1014 19:27:01.725291 20608 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I1014 19:27:01.726794 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.726830 20660 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.727069 20660 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.727068 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:01.825238 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:01.825356 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (starting election)
I1014 19:27:01.825454 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 453 FOLLOWER]: Advancing to term 454
I1014 19:27:01.827008 20490 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 453 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:01.827093 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:01.827159 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 453 FOLLOWER]: Advancing to term 454
I1014 19:27:01.827965 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.828527 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 454 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.828699 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 454 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.829357 20490 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.829838 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 454 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.829980 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 454 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:01.830772 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 454 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:01.830994 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 454: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:01.831287 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 454 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.831367 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 454 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.831485 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 454 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.831568 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 454: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:01.831919 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 454 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.832016 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 454 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 454: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:01.832056 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 454 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.832216 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 454 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 454 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.832291 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 454 election: Election decided. Result: candidate lost.
I1014 19:27:01.832792 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 454 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 454: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:01.832947 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 454 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 454 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:01.833021 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 454 election: Election decided. Result: candidate lost.
I1014 19:27:01.833230 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:01.833286 19920 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Leader election lost for term 454. Reason: could not achieve majority
I1014 19:27:01.833359 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:01.833489 20490 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Leader election lost for term 454. Reason: could not achieve majority
I1014 19:27:01.833974 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 453 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.834225 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:01.834406 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 453. Current term is 454. Ops: []
I1014 19:27:01.834906 20607 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 453.296, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I1014 19:27:01.835157 20607 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Stepping down as leader of term 453
I1014 19:27:01.835198 20607 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:01.835361 20607 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 453.296, Last appended by leader: 296, Current term: 453, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.835606 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:01.835645 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:01.835762 20607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 453 FOLLOWER]: Advancing to term 454
I1014 19:27:01.944576 19920 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:01.944665 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.203s (starting election)
I1014 19:27:01.944721 19920 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 454 FOLLOWER]: Advancing to term 455
I1014 19:27:01.947010 19920 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.947618 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 455 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.947799 19920 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 455 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.948495 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 455 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:01.948536 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 455 candidate_status { last_received { term: 453 index: 296 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:01.948645 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 454 FOLLOWER]: Advancing to term 455
I1014 19:27:01.948709 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:01.948702 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 454 FOLLOWER]: Advancing to term 455
I1014 19:27:01.948791 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.196s (vote granted)
I1014 19:27:01.950850 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 455 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 455.
I1014 19:27:01.950956 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 455 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 455.
I1014 19:27:01.951290 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 455 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:01.951349 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 455 election: Election decided. Result: candidate won.
I1014 19:27:01.951603 19920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.166s (election complete)
I1014 19:27:01.951660 19920 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 FOLLOWER]: Leader election won for term 455
I1014 19:27:01.951898 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 455 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:01.952103 19920 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:01.952306 19920 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 453.296, Last appended by leader: 296, Current term: 455, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:01.953794 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:01.954046 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:27:02.054975 15726 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.045s	user 0.002s	sys 0.000s
I1014 19:27:02.055043 15726 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:02.055971 15726 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:02.057073 15726 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 449 vs 455: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 455 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.057348 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 455 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.057582 15726 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 455 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.058714 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 455 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 453 index: 296. Preceding OpId from leader: term: 455 index: 297. (index mismatch)
I1014 19:27:02.059293 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I1014 19:27:02.060508 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.060703 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.061535 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 455 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 453 index: 296. Preceding OpId from leader: term: 455 index: 297. (index mismatch)
I1014 19:27:02.062216 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I1014 19:27:02.063138 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.063318 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.063575 20607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.063855 20607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.063979 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.064167 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.064455 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.064642 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.163462 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 455 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:02.163583 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:02.163645 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 455 FOLLOWER]: Advancing to term 456
I1014 19:27:02.164891 20490 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 455 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:02.164983 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:27:02.165052 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 455 FOLLOWER]: Advancing to term 456
I1014 19:27:02.165942 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.166457 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 456 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:02.167182 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 456 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.167203 20490 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 456 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.167699 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 456 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:02.167860 20490 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 456 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.167898 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 456 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:02.168092 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 456 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.168226 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 456 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:02.168397 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 456 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 456: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:02.168275 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 455 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.168555 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.168869 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 455. Current term is 456. Ops: []
I1014 19:27:02.168926 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 456 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 456 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 456: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:02.169124 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 456 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:02.169142 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 456 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:02.169267 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 456 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.169301 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 456: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:02.169301 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 456 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 456 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.169356 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 456 election: Election decided. Result: candidate lost.
I1014 19:27:02.169692 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 456 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 456 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.169725 20607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:02.169780 20607 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Leader election lost for term 456. Reason: could not achieve majority
I1014 19:27:02.170148 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 456 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 456: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:02.170135 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 455.297, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I1014 19:27:02.170234 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 456 election: Election decided. Result: candidate lost.
I1014 19:27:02.170330 20874 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Stepping down as leader of term 455
I1014 19:27:02.170373 20874 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:02.170498 20490 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:02.170568 20490 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 456 FOLLOWER]: Leader election lost for term 456. Reason: could not achieve majority
I1014 19:27:02.170574 20874 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 455.297, Last appended by leader: 297, Current term: 455, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.170779 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:02.170822 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:02.170953 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 455 FOLLOWER]: Advancing to term 456
I1014 19:27:02.251729 20885 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 456 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:02.251902 20885 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:27:02.251989 20885 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 456 FOLLOWER]: Advancing to term 457
I1014 19:27:02.254509 20885 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.255122 20885 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 457 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:02.255327 20885 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 457 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:02.256054 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 457 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:02.256088 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 457 candidate_status { last_received { term: 455 index: 297 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:02.256225 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 456 FOLLOWER]: Advancing to term 457
I1014 19:27:02.256263 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 456 FOLLOWER]: Advancing to term 457
I1014 19:27:02.256294 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.210s (vote granted)
I1014 19:27:02.256345 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.214s (vote granted)
I1014 19:27:02.258441 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 457.
I1014 19:27:02.258496 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 457 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 457.
I1014 19:27:02.258955 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 457 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:02.259034 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 457 election: Election decided. Result: candidate won.
I1014 19:27:02.259297 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 457 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:02.259284 20885 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:02.259363 20885 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 FOLLOWER]: Leader election won for term 457
I1014 19:27:02.259491 20885 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:02.259739 20885 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 455.297, Last appended by leader: 297, Current term: 457, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.260816 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.261003 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.368067 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 455 index: 297. Preceding OpId from leader: term: 457 index: 298. (index mismatch)
I1014 19:27:02.368757 20885 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I1014 19:27:02.370051 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.370260 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.371282 19920 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.371554 19920 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.371991 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 457 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 455 index: 297. Preceding OpId from leader: term: 457 index: 298. (index mismatch)
I1014 19:27:02.372081 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.372267 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.372957 19920 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I1014 19:27:02.376674 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.376911 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.377274 21008 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.377527 21008 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.474982 21008 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 457 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:02.475069 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:02.475139 21008 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 457 FOLLOWER]: Advancing to term 458
I1014 19:27:02.477217 21008 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.477813 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 458 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:02.478603 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 458 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:02.478790 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 458 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.479107 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 458 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.479583 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 458 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 458 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.479815 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 458 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:02.480008 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 458 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.480389 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 458 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 458 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.480468 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 458 election: Election decided. Result: candidate lost.
I1014 19:27:02.480737 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:02.480844 21008 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Leader election lost for term 458. Reason: could not achieve majority
I1014 19:27:02.494570 20885 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 457 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.494866 20885 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.495213 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 457. Current term is 458. Ops: []
I1014 19:27:02.495803 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 457.298, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I1014 19:27:02.496013 20874 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 LEADER]: Stepping down as leader of term 457
I1014 19:27:02.496049 20874 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:02.496250 20874 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 457.298, Last appended by leader: 298, Current term: 457, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.496438 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:02.496502 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:02.496619 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 457 FOLLOWER]: Advancing to term 458
I1014 19:27:02.563557 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:02.563635 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (starting election)
I1014 19:27:02.563690 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 457 FOLLOWER]: Advancing to term 458
I1014 19:27:02.565945 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.566408 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 458 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:02.566577 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 458 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.567414 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 458 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:02.567600 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
W1014 19:27:02.568706 20771 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.232s	user 0.002s	sys 0.000s
I1014 19:27:02.568738 20771 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:02.569661 20771 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:02.569718 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 458 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 458.
I1014 19:27:02.570133 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 458 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.570194 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 458 election: Election decided. Result: candidate won.
I1014 19:27:02.570653 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:02.570708 20772 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 FOLLOWER]: Leader election won for term 458
I1014 19:27:02.570777 20772 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:02.570937 20772 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 457.298, Last appended by leader: 298, Current term: 458, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.571017 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 458 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:02.571218 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 458: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:02.571609 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 458 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 458: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:02.571843 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 458 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.572062 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.572144 20771 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 451 vs 458: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.572274 20771 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:02.573978 20771 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:27:02.577813 19783 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.607s	user 0.002s	sys 0.000s
I1014 19:27:02.577858 19783 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:02.578673 19783 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:02.579811 19783 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 448 vs 458: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.580103 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.580338 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.580679 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.580937 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.581172 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.581413 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:02.581681 19783 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 458 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.584683 21008 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:02.584767 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.220s (starting election)
I1014 19:27:02.584830 21008 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 458 FOLLOWER]: Advancing to term 459
I1014 19:27:02.587057 21008 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.587630 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 459 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:02.588469 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 459 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:02.588670 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 459 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.588901 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 459 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.589215 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 459 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 459 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:02.589876 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 459 candidate_status { last_received { term: 457 index: 298 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:02.590023 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 458 FOLLOWER]: Advancing to term 459
I1014 19:27:02.590090 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (vote granted)
I1014 19:27:02.592279 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 459 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 459.
I1014 19:27:02.592664 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 459 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:02.592782 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 459 election: Election decided. Result: candidate won.
I1014 19:27:02.593010 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.208s (election complete)
I1014 19:27:02.593078 21008 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 FOLLOWER]: Leader election won for term 459
I1014 19:27:02.593236 21008 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:02.593462 21008 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 457.298, Last appended by leader: 298, Current term: 459, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.594651 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.594856 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.609912 21065 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:02.611961 21065 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:02.658447 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 459 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 458. Current term is 459. Ops: []
I1014 19:27:02.659046 20607 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I1014 19:27:02.659265 20607 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 LEADER]: Stepping down as leader of term 458
I1014 19:27:02.659296 20607 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:02.659442 20607 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 458.299, Last appended by leader: 299, Current term: 458, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:02.659641 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:02.659672 20607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:02.659757 20607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 458 FOLLOWER]: Advancing to term 459
I1014 19:27:02.705621 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 458 index: 299. Preceding OpId from leader: term: 459 index: 299. (term mismatch)
I1014 19:27:02.705672 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 298
I1014 19:27:02.705689 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 458.299
I1014 19:27:02.705723 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:02.706342 20490 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I1014 19:27:02.707587 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.707780 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.708947 21008 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.709272 21008 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:02.709434 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 459 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 457 index: 298. Preceding OpId from leader: term: 459 index: 299. (index mismatch)
I1014 19:27:02.709995 21008 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I1014 19:27:02.710156 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.710330 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.711418 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.711694 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.711807 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.712090 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:02.795920 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 459 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:02.796160 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.082372 20874 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 459 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:03.082464 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:03.082511 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 459 FOLLOWER]: Advancing to term 460
I1014 19:27:03.084707 20874 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.085160 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 460 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.085306 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 460 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.085618 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 459. Current term is 460. Ops: []
I1014 19:27:03.085950 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 460 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.085958 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 460 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.086140 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 460 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.086184 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 460 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.086280 20490 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 459.299, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I1014 19:27:03.086599 21008 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 LEADER]: Stepping down as leader of term 459
I1014 19:27:03.086666 21008 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:03.086681 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 460 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 460 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.086853 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 460 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 460 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.086905 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 460 election: Election decided. Result: candidate lost.
I1014 19:27:03.086911 21008 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 459.299, Last appended by leader: 299, Current term: 459, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.087085 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:03.087126 21008 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:03.087138 20874 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Leader election lost for term 460. Reason: could not achieve majority
I1014 19:27:03.087186 21008 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:03.087329 21008 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 459 FOLLOWER]: Advancing to term 460
I1014 19:27:03.175299 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:03.175392 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:03.175449 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 459 FOLLOWER]: Advancing to term 460
I1014 19:27:03.175662 21008 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 460 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:03.175747 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:03.175812 21008 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 460 FOLLOWER]: Advancing to term 461
I1014 19:27:03.176964 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 460 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.177325 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 460 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.177448 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 460 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.177974 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 460 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.177978 21008 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.178419 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 461 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.178632 21008 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 461 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.178654 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 460 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.178866 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 460: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:03.179217 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 461 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.179343 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Advancing to term 461
I1014 19:27:03.179407 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.185s (vote granted)
I1014 19:27:03.179356 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 461 candidate_status { last_received { term: 459 index: 299 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.179503 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 460 FOLLOWER]: Advancing to term 461
I1014 19:27:03.179502 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 460 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 460 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 460: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:03.179558 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:03.179662 28826 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 460. Current term is 461.
W1014 19:27:03.180169 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 460 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 460. Current term is 461.
I1014 19:27:03.180196 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 460 election: Cancelling election due to peer responding with higher term
I1014 19:27:03.181560 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 461.
I1014 19:27:03.181597 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 461 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 461.
I1014 19:27:03.181675 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:03.181728 20772 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 461 FOLLOWER]: Leader election lost for term 460. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 460. Current term is 461.
I1014 19:27:03.181895 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 461 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.181999 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 461 election: Election decided. Result: candidate won.
I1014 19:27:03.182129 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 461 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.182271 21008 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:03.182343 21008 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Leader election won for term 461
I1014 19:27:03.182490 21008 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:03.182693 21008 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 459.299, Last appended by leader: 299, Current term: 461, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.183787 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.183990 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.266381 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 461 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 459 index: 299. Preceding OpId from leader: term: 461 index: 300. (index mismatch)
I1014 19:27:03.266952 20490 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I1014 19:27:03.268122 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.268297 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.275764 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.275950 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.276940 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.277173 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.277407 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 459 index: 299. Preceding OpId from leader: term: 461 index: 300. (index mismatch)
I1014 19:27:03.277926 21008 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I1014 19:27:03.280663 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.280881 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.281450 21420 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.281721 21420 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.366909 21008 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 461 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.367126 21008 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.370072 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 461 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:03.370141 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:03.370187 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 461 FOLLOWER]: Advancing to term 462
I1014 19:27:03.372254 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.372743 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 462 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.372886 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 462 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.373487 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 462 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.373682 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 462 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.373731 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 462 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.373931 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 462 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.374075 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 462 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 462 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.374850 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 462 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 462 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.374915 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 462 election: Election decided. Result: candidate lost.
I1014 19:27:03.375468 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:03.375527 20772 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Leader election lost for term 462. Reason: could not achieve majority
I1014 19:27:03.385884 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 461. Current term is 462. Ops: []
I1014 19:27:03.386425 20490 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 461.300, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I1014 19:27:03.386718 20490 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 LEADER]: Stepping down as leader of term 461
I1014 19:27:03.386772 20490 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:03.386970 20490 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 461.300, Last appended by leader: 300, Current term: 461, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.387140 20490 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:03.387187 20490 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:03.387311 20490 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 461 FOLLOWER]: Advancing to term 462
I1014 19:27:03.468986 21420 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:03.469075 21420 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:03.469148 21420 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 461 FOLLOWER]: Advancing to term 462
I1014 19:27:03.471395 21420 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.471948 21420 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 462 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.472131 21420 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 462 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.476094 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 462 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.476161 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 462 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.476290 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:03.476372 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 462: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:03.476794 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 462 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 462: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:03.478381 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 462 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 462.
I1014 19:27:03.478696 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 462 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.478744 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 462 election: Election decided. Result: candidate won.
I1014 19:27:03.478930 21420 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:03.478991 21420 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 FOLLOWER]: Leader election won for term 462
I1014 19:27:03.479074 21420 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:03.479288 21420 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 461.300, Last appended by leader: 300, Current term: 462, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.480300 21420 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 462 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.480536 21420 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.486279 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:03.488570 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:03.519321 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:03.519404 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:03.519481 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 462 FOLLOWER]: Advancing to term 463
I1014 19:27:03.521653 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.522142 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 463 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.522588 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 463 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.522922 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 463 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.523085 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 462 FOLLOWER]: Advancing to term 463
I1014 19:27:03.523167 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:03.525260 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 463 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 463.
I1014 19:27:03.525863 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 463 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.525827 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 463 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.525918 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 463 election: Election decided. Result: candidate won.
I1014 19:27:03.526011 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 463 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.526175 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:03.526387 20772 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 FOLLOWER]: Leader election won for term 463
I1014 19:27:03.526397 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 463 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 463 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.538234 20772 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:03.538794 20772 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 461.300, Last appended by leader: 300, Current term: 463, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.539989 21487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 463 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.540280 21487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.575225 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 463 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 462. Current term is 463. Ops: []
I1014 19:27:03.575801 21420 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I1014 19:27:03.576081 21420 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 LEADER]: Stepping down as leader of term 462
I1014 19:27:03.576131 21420 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:03.576326 21420 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 462.301, Last appended by leader: 301, Current term: 462, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.576544 21420 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:03.576591 21420 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:03.576714 21420 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 462 FOLLOWER]: Advancing to term 463
I1014 19:27:03.649554 21138 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 463 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:03.649658 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:27:03.649744 21138 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 463 FOLLOWER]: Advancing to term 464
I1014 19:27:03.651301 21138 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 464 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.651818 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 464 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.652025 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 464 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.652550 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 464 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.652602 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 464 candidate_status { last_received { term: 461 index: 300 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.652715 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 464 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.652776 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 463 FOLLOWER]: Advancing to term 464
I1014 19:27:03.655098 29011 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 464 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 464 because replica has last-logged OpId of term: 462 index: 301, which is greater than that of the candidate, which has last-logged OpId of term: 461 index: 300.
I1014 19:27:03.655534 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 464 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 464 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 464 because replica has last-logged OpId of term: 462 index: 301, which is greater than that of the candidate, which has last-logged OpId of term: 461 index: 300.
I1014 19:27:03.655676 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 464 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 464 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:03.655756 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 464 election: Election decided. Result: candidate lost.
I1014 19:27:03.656136 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:03.656241 21138 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 464 FOLLOWER]: Leader election lost for term 464. Reason: could not achieve majority
I1014 19:27:03.656430 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 464 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 463. Current term is 464. Ops: []
I1014 19:27:03.656986 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 464 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 463. Current term is 464. Ops: []
I1014 19:27:03.657102 21487 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I1014 19:27:03.657305 21487 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 LEADER]: Stepping down as leader of term 463
I1014 19:27:03.657346 21487 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:03.657522 21487 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 463.301, Last appended by leader: 301, Current term: 463, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.657693 21487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:03.657727 21487 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:03.657802 21487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 463 FOLLOWER]: Advancing to term 464
I1014 19:27:03.662816 20874 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 464 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:03.662878 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:03.662936 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 464 FOLLOWER]: Advancing to term 465
I1014 19:27:03.664966 20874 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.665434 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 465 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.665606 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 465 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.666225 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 465 candidate_status { last_received { term: 462 index: 301 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.666260 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 465 candidate_status { last_received { term: 462 index: 301 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.666368 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 464 FOLLOWER]: Advancing to term 465
I1014 19:27:03.666429 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 464 FOLLOWER]: Advancing to term 465
I1014 19:27:03.666508 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:27:03.668416 28639 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 465 because replica has last-logged OpId of term: 463 index: 301, which is greater than that of the candidate, which has last-logged OpId of term: 462 index: 301.
I1014 19:27:03.668566 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 465 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 465.
I1014 19:27:03.668879 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 465 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.668933 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 465 election: Election decided. Result: candidate won.
I1014 19:27:03.669111 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:03.669174 20874 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 FOLLOWER]: Leader election won for term 465
I1014 19:27:03.669205 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 465 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 465 because replica has last-logged OpId of term: 463 index: 301, which is greater than that of the candidate, which has last-logged OpId of term: 462 index: 301.
I1014 19:27:03.669248 20874 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:03.669422 20874 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 462.301, Last appended by leader: 301, Current term: 465, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.670300 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.670562 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.747613 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 463 index: 301. Preceding OpId from leader: term: 465 index: 302. (index mismatch)
I1014 19:27:03.748198 21420 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 300, Time since last communication: 0.000s
I1014 19:27:03.748359 21420 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 463.301 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 300
I1014 19:27:03.748955 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 300
I1014 19:27:03.748982 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 463.301
I1014 19:27:03.749022 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:03.749785 21487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.749999 21487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.750986 21420 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.751207 21420 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.751655 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 465 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 461 index: 300. Preceding OpId from leader: term: 465 index: 302. (index mismatch)
I1014 19:27:03.751832 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.752065 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.752159 21487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.752310 21487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.752969 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 300, Time since last communication: 0.000s
I1014 19:27:03.754693 21008 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.754986 21008 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.754894 20490 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.755156 20490 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.755157 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 465 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.755364 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.852505 21487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:03.852602 21487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:03.852660 21487 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 465 FOLLOWER]: Advancing to term 466
I1014 19:27:03.854488 21138 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 465 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:03.854571 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:03.854629 21138 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 465 FOLLOWER]: Advancing to term 466
I1014 19:27:03.854941 21487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.855245 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 465. Current term is 466. Ops: []
I1014 19:27:03.855451 21487 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 466 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:03.856076 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 465.302, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I1014 19:27:03.856310 21600 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 LEADER]: Stepping down as leader of term 465
I1014 19:27:03.856354 21600 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:03.856463 21487 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 466 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.856580 21600 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 465.302, Last appended by leader: 302, Current term: 465, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.856696 21138 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.856837 21600 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:03.856879 21600 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:03.857010 21600 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 465 FOLLOWER]: Advancing to term 466
I1014 19:27:03.857123 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 466 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:03.857277 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 466 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.857583 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 466 candidate_status { last_received { term: 465 index: 302 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.858232 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 466 candidate_status { last_received { term: 465 index: 302 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:03.858273 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 466 candidate_status { last_received { term: 465 index: 302 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:03.858387 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 466: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:03.858783 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 466 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 466: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:03.859372 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (vote granted)
I1014 19:27:03.859972 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 466 candidate_status { last_received { term: 465 index: 302 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:03.860155 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 466: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:03.860513 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 466 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 466: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:03.861395 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 466.
I1014 19:27:03.862637 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 466 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:03.862691 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 466 election: Election decided. Result: candidate won.
I1014 19:27:03.863028 21487 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:03.863086 21487 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Leader election won for term 466
I1014 19:27:03.863173 21487 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:03.863363 21487 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 465.302, Last appended by leader: 302, Current term: 466, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:03.864331 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.864536 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.865633 29010 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 466 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:03.866099 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 466 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Service unavailable: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 466 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:03.866186 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 466 election: Election decided. Result: candidate lost.
I1014 19:27:03.883788 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (election complete)
I1014 19:27:03.883877 21138 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Leader election lost for term 466. Reason: could not achieve majority
I1014 19:27:03.950484 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 465 index: 302. Preceding OpId from leader: term: 466 index: 303. (index mismatch)
I1014 19:27:03.950933 20772 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I1014 19:27:03.952054 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.952311 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.953116 21487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.953426 21487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:03.954352 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 465 index: 302. Preceding OpId from leader: term: 466 index: 303. (index mismatch)
I1014 19:27:03.954776 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.954972 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.955209 21689 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I1014 19:27:03.956745 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.956929 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:03.957072 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:03.957232 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.055745 21138 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:04.055832 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (starting election)
I1014 19:27:04.055886 21138 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 466 FOLLOWER]: Advancing to term 467
I1014 19:27:04.056533 21600 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:04.056604 21600 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:04.056659 21600 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 466 FOLLOWER]: Advancing to term 467
I1014 19:27:04.057920 21138 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.058007 21689 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 466 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.058239 21689 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.058468 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 467 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.058368 21600 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.058614 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 467 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.058662 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 466. Current term is 467. Ops: []
I1014 19:27:04.058806 21600 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 467 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.058938 21600 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 467 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.059336 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 467 candidate_status { last_received { term: 466 index: 303 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:04.059528 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 467: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:04.059561 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 467 candidate_status { last_received { term: 466 index: 303 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:04.059659 20772 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 466.303, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I1014 19:27:04.059895 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 466. Current term is 467. Ops: []
I1014 19:27:04.059923 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 467 candidate_status { last_received { term: 466 index: 303 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:04.060103 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 467 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 467: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:04.060139 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 467: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:04.060256 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 467 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.060326 21689 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 LEADER]: Stepping down as leader of term 466
I1014 19:27:04.060367 21689 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:04.060681 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 467 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 467 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.060564 21689 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 466.303, Last appended by leader: 303, Current term: 466, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.060739 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 467 election: Election decided. Result: candidate lost.
I1014 19:27:04.060739 21689 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:04.060727 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 467 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 467: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:04.060778 21689 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:04.060801 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 467 candidate_status { last_received { term: 466 index: 303 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:04.060866 21689 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 466 FOLLOWER]: Advancing to term 467
I1014 19:27:04.061094 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:04.061148 21138 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Leader election lost for term 467. Reason: could not achieve majority
I1014 19:27:04.062955 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (vote granted)
W1014 19:27:04.062950 20772 consensus_queue.cc:1042] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cbc42c7d5db44e00883549e254bdd22e" responder_term: 467 status { last_received { term: 466 index: 303 } last_committed_idx: 303 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 466. Current term is 467. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I1014 19:27:04.064905 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 467 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 467.
I1014 19:27:04.065294 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 467 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.065351 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 467 election: Election decided. Result: candidate won.
I1014 19:27:04.065654 21600 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:04.065719 21600 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Leader election won for term 467
I1014 19:27:04.065809 21600 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:04.065968 21600 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 466.303, Last appended by leader: 303, Current term: 467, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.066881 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.067155 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.128049 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:04.128880 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:04.131372 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 466 index: 303. Preceding OpId from leader: term: 467 index: 305. (index mismatch)
I1014 19:27:04.131372 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 467 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 466 index: 303. Preceding OpId from leader: term: 467 index: 305. (index mismatch)
I1014 19:27:04.132025 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I1014 19:27:04.132200 21600 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I1014 19:27:04.133864 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.134058 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.134376 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.134627 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.136049 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.136265 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.137197 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.137415 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.140125 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.141808 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.142601 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:04.145107 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:04.145395 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 467 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.145615 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.333338 21138 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:04.333439 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:04.333500 21138 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 467 FOLLOWER]: Advancing to term 468
I1014 19:27:04.335850 21138 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.336156 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 467. Current term is 468. Ops: []
I1014 19:27:04.336355 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 468 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.336488 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 468 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.336992 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 468 candidate_status { last_received { term: 467 index: 305 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:04.337169 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 468 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.337244 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 467.305, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I1014 19:27:04.337545 20874 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 LEADER]: Stepping down as leader of term 467
I1014 19:27:04.337596 20874 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:04.337781 20874 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 467.305, Last appended by leader: 305, Current term: 467, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.337985 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:04.338032 20874 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:04.338160 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 467 FOLLOWER]: Advancing to term 468
I1014 19:27:04.340488 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 468 candidate_status { last_received { term: 467 index: 305 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:04.340641 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:04.342682 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 468.
I1014 19:27:04.343148 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 468 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.343219 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 468 election: Election decided. Result: candidate won.
I1014 19:27:04.343528 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 468 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 467 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 468 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.343749 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:04.343816 21138 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 FOLLOWER]: Leader election won for term 468
W1014 19:27:04.345031 21065 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.733s	user 0.001s	sys 0.000s
I1014 19:27:04.345069 21065 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:04.345922 21065 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:04.353261 21138 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:04.353549 21138 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 467.305, Last appended by leader: 305, Current term: 468, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.354722 21920 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.355218 21920 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.356765 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 467 index: 305. Preceding OpId from leader: term: 468 index: 307. (index mismatch)
I1014 19:27:04.356801 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 467 FOLLOWER]: Advancing to term 468
I1014 19:27:04.357293 21920 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I1014 19:27:04.359063 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 468 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 467 index: 305. Preceding OpId from leader: term: 468 index: 307. (index mismatch)
I1014 19:27:04.358888 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.359141 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.361064 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.361248 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.362043 21920 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I1014 19:27:04.367120 21065 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:04.368700 21689 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.368970 21689 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.369040 21065 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:04.368927 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.369159 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.369412 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.369654 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.453403 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.453595 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:27:04.457384 20771 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.883s	user 0.001s	sys 0.000s
I1014 19:27:04.457427 20771 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:04.458086 20771 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:04.458920 20771 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 458 vs 468: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.459151 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.459359 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.459594 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.459864 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.460613 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.460871 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.461129 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.461395 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:04.461684 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 468 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.554921 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 468 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:04.555002 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:04.555052 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 468 FOLLOWER]: Advancing to term 469
I1014 19:27:04.557201 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.557716 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 469 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.557868 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 469 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.558976 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 469 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:04.559164 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 469 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.559545 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 469 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:04.559615 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 469 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 469 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.559751 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 469 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.560134 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 469 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 469 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.560199 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 469 election: Election decided. Result: candidate lost.
I1014 19:27:04.561875 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:04.561942 20772 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Leader election lost for term 469. Reason: could not achieve majority
I1014 19:27:04.566051 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 468. Current term is 469. Ops: []
I1014 19:27:04.566594 21138 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 468.307, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I1014 19:27:04.566810 21921 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 LEADER]: Stepping down as leader of term 468
I1014 19:27:04.566859 21921 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:04.567034 21921 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 468.307, Last appended by leader: 307, Current term: 468, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.567272 21921 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:04.567307 21921 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:04.567418 21921 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 468 FOLLOWER]: Advancing to term 469
I1014 19:27:04.643381 21600 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:04.643512 21600 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:04.643599 21600 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 468 FOLLOWER]: Advancing to term 469
I1014 19:27:04.645170 21600 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.645648 21600 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 469 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.645802 21600 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 469 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.646456 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 469 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:04.646646 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 469: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:04.646698 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 469 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:04.646867 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (vote granted)
I1014 19:27:04.647028 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 469 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 469: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:04.648813 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 469 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 469.
I1014 19:27:04.649149 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 469 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.649214 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 469 election: Election decided. Result: candidate won.
I1014 19:27:04.649454 21600 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:04.649575 21600 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 FOLLOWER]: Leader election won for term 469
I1014 19:27:04.649714 21600 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:04.649916 21600 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 468.307, Last appended by leader: 307, Current term: 469, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.650889 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 469 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.651121 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.680194 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:04.680264 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.191s (starting election)
I1014 19:27:04.680313 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 469 FOLLOWER]: Advancing to term 470
I1014 19:27:04.682409 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.682966 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 470 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.683130 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 470 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.683827 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 470 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:04.684048 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 470 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.684376 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 470 candidate_status { last_received { term: 468 index: 307 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:04.684429 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 470 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 470 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.684540 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 469 FOLLOWER]: Advancing to term 470
I1014 19:27:04.684614 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:04.686600 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 470 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 470.
I1014 19:27:04.687050 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 470 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:04.687103 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 470 election: Election decided. Result: candidate won.
I1014 19:27:04.687649 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:04.687711 20772 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 FOLLOWER]: Leader election won for term 470
I1014 19:27:04.687790 20772 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:04.687980 20772 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 468.307, Last appended by leader: 307, Current term: 470, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.689401 21689 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.689622 21689 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.690155 20771 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:04.692019 20771 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:04.750494 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 470 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 469. Current term is 470. Ops: []
I1014 19:27:04.751147 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I1014 19:27:04.751410 21600 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 LEADER]: Stepping down as leader of term 469
I1014 19:27:04.751464 21600 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:04.751711 21600 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 307, Committed index: 307, Last appended: 469.308, Last appended by leader: 308, Current term: 469, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.751922 21600 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:04.751982 21600 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:04.752121 21600 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 469 FOLLOWER]: Advancing to term 470
I1014 19:27:04.774341 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 470 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 469 index: 308. Preceding OpId from leader: term: 470 index: 308. (term mismatch)
I1014 19:27:04.774399 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 307
I1014 19:27:04.774437 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 469.308
I1014 19:27:04.774502 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 470 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:04.775295 21689 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I1014 19:27:04.776902 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.777753 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.777819 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.778002 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.778632 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 470 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 468 index: 307. Preceding OpId from leader: term: 470 index: 308. (index mismatch)
I1014 19:27:04.779242 21689 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 308, Last known committed idx: 307, Time since last communication: 0.000s
I1014 19:27:04.779336 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.779907 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.780989 21921 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.781009 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.781185 21921 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.781203 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:04.863791 21689 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 470 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.863991 21689 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:04.965620 21138 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 470 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:04.965698 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (starting election)
I1014 19:27:04.965757 21138 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 470 FOLLOWER]: Advancing to term 471
I1014 19:27:04.967818 21138 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.968308 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 471 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:04.968350 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 470. Current term is 471. Ops: []
I1014 19:27:04.968482 21138 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 471 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.968905 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 471 candidate_status { last_received { term: 470 index: 308 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:04.969066 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 471 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.969095 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 471 candidate_status { last_received { term: 470 index: 308 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:04.969113 20772 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 470.308, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I1014 19:27:04.969282 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 470 FOLLOWER]: Advancing to term 471
I1014 19:27:04.969349 20772 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 LEADER]: Stepping down as leader of term 470
I1014 19:27:04.969385 20772 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:04.969374 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:04.969501 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 471 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 471 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:04.969573 20772 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 470.308, Last appended by leader: 308, Current term: 470, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.969746 20772 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:04.969796 20772 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:04.969915 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 470 FOLLOWER]: Advancing to term 471
I1014 19:27:04.971499 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 471 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 471.
I1014 19:27:04.971884 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 471 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:04.971933 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 471 election: Election decided. Result: candidate won.
I1014 19:27:04.972115 21138 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:04.972173 21138 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 FOLLOWER]: Leader election won for term 471
I1014 19:27:04.972251 21138 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:04.972417 21138 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 470.308, Last appended by leader: 308, Current term: 471, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:04.973853 21921 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:04.974026 21921 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.052256 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 471 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 470 index: 308. Preceding OpId from leader: term: 471 index: 309. (index mismatch)
I1014 19:27:05.053479 21921 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I1014 19:27:05.054826 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.055001 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.056735 21921 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.056913 21921 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.057183 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.057366 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.057595 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 471 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 470 index: 308. Preceding OpId from leader: term: 471 index: 309. (index mismatch)
I1014 19:27:05.058097 21138 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I1014 19:27:05.059300 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.059545 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.059661 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.059837 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.070768 21065 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:05.071596 21065 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:05.078001 21921 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 471 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.078204 21921 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.088634 21065 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:05.090354 21065 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:05.182113 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 471 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:05.182200 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:05.182260 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 471 FOLLOWER]: Advancing to term 472
I1014 19:27:05.182260 20874 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 471 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:05.182349 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:05.182413 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 471 FOLLOWER]: Advancing to term 472
I1014 19:27:05.184479 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.184615 20874 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 472 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.184978 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 472 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.185060 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 472 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:05.185470 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 472 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.185395 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 472 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:05.186072 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 472 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:05.186074 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 472 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:05.186254 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 472: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:05.186254 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 472 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.186547 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 472 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:05.186693 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 472 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.186738 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 472 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 472: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:05.186957 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 472 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 472 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.186883 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 472 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:05.187077 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 472 election: Election decided. Result: candidate lost.
I1014 19:27:05.187144 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 472 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 472: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:05.187348 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:05.187389 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 472 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 472 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.187474 20874 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 472 FOLLOWER]: Leader election lost for term 472. Reason: could not achieve majority
I1014 19:27:05.187594 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 472 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 472 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 472: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:05.187655 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 472 election: Election decided. Result: candidate lost.
I1014 19:27:05.187814 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:05.187873 20772 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Leader election lost for term 472. Reason: could not achieve majority
I1014 19:27:05.201486 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 471. Current term is 472. Ops: []
I1014 19:27:05.201977 22393 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 471.310, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I1014 19:27:05.202221 22393 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Stepping down as leader of term 471
I1014 19:27:05.202267 22393 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:05.202476 22393 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 471.310, Last appended by leader: 310, Current term: 471, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.202666 22393 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:05.202705 22393 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:05.202831 22393 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 471 FOLLOWER]: Advancing to term 472
I1014 19:27:05.280560 22394 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 472 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:05.280638 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (starting election)
I1014 19:27:05.280714 22394 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 472 FOLLOWER]: Advancing to term 473
I1014 19:27:05.282712 22394 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.283238 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 473 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:05.283416 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 473 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:05.284047 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 473 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:05.284188 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 472 FOLLOWER]: Advancing to term 473
I1014 19:27:05.284255 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.217s (vote granted)
I1014 19:27:05.284165 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 473 candidate_status { last_received { term: 471 index: 310 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:05.284356 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 472 FOLLOWER]: Advancing to term 473
I1014 19:27:05.284452 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:05.285701 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 473.
I1014 19:27:05.286065 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 473 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:05.286123 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 473 election: Election decided. Result: candidate won.
I1014 19:27:05.286388 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:05.286463 22394 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 FOLLOWER]: Leader election won for term 473
I1014 19:27:05.286541 22394 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:05.286676 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 473 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 473.
I1014 19:27:05.287137 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 473 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:05.287477 22394 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 310, Committed index: 310, Last appended: 471.310, Last appended by leader: 310, Current term: 473, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.288740 22393 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.288914 22393 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.356150 21065 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:05.356986 21065 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:05.359175 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 473 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 471 index: 310. Preceding OpId from leader: term: 473 index: 312. (index mismatch)
I1014 19:27:05.359179 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 471 index: 310. Preceding OpId from leader: term: 473 index: 312. (index mismatch)
I1014 19:27:05.359668 22393 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I1014 19:27:05.359807 22394 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 311, Last known committed idx: 310, Time since last communication: 0.000s
I1014 19:27:05.361832 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.362044 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.362677 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.362884 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.364339 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.364549 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.364713 21138 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.364863 21138 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.364823 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.365038 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.366286 20772 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.366472 20772 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.366762 21065 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:05.368887 21065 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:27:05.385057 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.240s	user 0.002s	sys 0.000s
I1014 19:27:05.385128 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:05.386075 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:05.396519 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 467 vs 473: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.396807 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.397022 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.397238 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.397460 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.397661 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 473 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.633656 20874 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 473 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:05.633765 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:05.633869 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 473 FOLLOWER]: Advancing to term 474
I1014 19:27:05.636051 20874 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.636600 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 474 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:05.636765 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 474 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.637413 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 474 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:05.637481 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 474 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:05.637627 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 474 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.637670 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 474 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.638085 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 474 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 474 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.638231 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 474 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 474 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.638283 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 474 election: Election decided. Result: candidate lost.
I1014 19:27:05.638475 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:05.638535 20874 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Leader election lost for term 474. Reason: could not achieve majority
I1014 19:27:05.641590 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 473. Current term is 474. Ops: []
I1014 19:27:05.642006 21138 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 473.312, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I1014 19:27:05.642202 22394 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 LEADER]: Stepping down as leader of term 473
I1014 19:27:05.642243 22394 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:05.642401 22394 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 473.312, Last appended by leader: 312, Current term: 473, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.642558 22394 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:05.642585 22394 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:05.642662 22394 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 473 FOLLOWER]: Advancing to term 474
I1014 19:27:05.731235 20772 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:05.731315 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:05.731364 20772 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 473 FOLLOWER]: Advancing to term 474
I1014 19:27:05.733393 20772 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.733889 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 474 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.734052 20772 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 474 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:05.734841 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 474 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:05.734843 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 474 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:05.735015 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (vote granted)
I1014 19:27:05.735100 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 474: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:05.735550 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 474 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 474: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:05.736984 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 474 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 474.
I1014 19:27:05.737303 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 474 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.737349 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 474 election: Election decided. Result: candidate won.
I1014 19:27:05.737635 20772 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:05.737684 20772 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 FOLLOWER]: Leader election won for term 474
I1014 19:27:05.738255 20772 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:05.738466 20772 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 473.312, Last appended by leader: 312, Current term: 474, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.739624 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 474 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.739856 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.767690 20874 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:05.767773 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:05.767825 20874 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 474 FOLLOWER]: Advancing to term 475
I1014 19:27:05.769979 20874 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.770470 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 475 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:05.770624 20874 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 475 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.771167 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 475 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:05.771167 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 475 candidate_status { last_received { term: 473 index: 312 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:05.771329 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 474 FOLLOWER]: Advancing to term 475
I1014 19:27:05.771361 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 475 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.771406 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (vote granted)
I1014 19:27:05.771709 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 475 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 475 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:05.772828 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 475 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 475.
I1014 19:27:05.773155 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 475 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:05.773210 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 475 election: Election decided. Result: candidate won.
I1014 19:27:05.773391 20874 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.180s (election complete)
I1014 19:27:05.773469 20874 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 FOLLOWER]: Leader election won for term 475
I1014 19:27:05.773701 20874 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:05.773932 20874 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 473.312, Last appended by leader: 312, Current term: 475, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.774870 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.775048 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.775244 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:05.776944 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:27:05.777588 20771 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.086s	user 0.002s	sys 0.000s
I1014 19:27:05.777629 20771 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:05.778422 20771 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:05.780683 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 475 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 474. Current term is 475. Ops: []
I1014 19:27:05.780802 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 474. Current term is 475. Ops: []
I1014 19:27:05.781385 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I1014 19:27:05.781795 22681 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 475.313 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 312
I1014 19:27:05.781893 20772 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I1014 19:27:05.782081 22681 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 LEADER]: Stepping down as leader of term 474
I1014 19:27:05.782119 22681 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:05.782281 22681 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 474.314, Last appended by leader: 314, Current term: 474, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:05.782524 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:05.782564 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:05.782676 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 474 FOLLOWER]: Advancing to term 475
I1014 19:27:05.855546 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 475 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 473 index: 312. Preceding OpId from leader: term: 475 index: 313. (index mismatch)
I1014 19:27:05.856027 22715 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I1014 19:27:05.857262 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.857476 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.858587 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.858783 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:05.859146 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.859308 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.859664 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 474 index: 314. Preceding OpId from leader: term: 475 index: 313. (term mismatch)
I1014 19:27:05.859709 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 312
I1014 19:27:05.859731 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 474.313
I1014 19:27:05.859771 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:05.859820 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted WRITE_OP operation due to leader change: 474.314
I1014 19:27:05.860131 20771 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 470 vs 475: Aborted: Transaction aborted by new leader
W1014 19:27:05.860409 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 475 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.860534 22715 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
W1014 19:27:05.860661 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 475 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:05.860849 20771 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 475 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.861682 22718 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.861863 22718 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.861713 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.861995 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:05.941089 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:05.941332 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.011924 21065 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:06.012878 21065 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:06.013942 21065 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 473 vs 475: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.014170 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.014341 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.014540 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.014703 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.014930 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.015138 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.015357 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.016005 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.016273 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.017423 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:06.017666 21065 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 475 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.040829 22394 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 475 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:06.040913 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:06.040957 22394 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 475 FOLLOWER]: Advancing to term 476
I1014 19:27:06.043036 22394 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.043568 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 476 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.043745 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 476 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.044418 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 476 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:06.044420 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 476 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.044631 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.044634 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.045027 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 476 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.045073 22681 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:06.045127 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:06.045177 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: Advancing to term 476
I1014 19:27:06.045258 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 476 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 475 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.045318 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 476 election: Election decided. Result: candidate lost.
I1014 19:27:06.045692 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:06.045753 22394 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Leader election lost for term 476. Reason: could not achieve majority
I1014 19:27:06.047224 22681 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 476 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.047721 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 476 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.047860 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 476 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.048223 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 475. Current term is 476. Ops: []
I1014 19:27:06.048269 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 476 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.048416 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 476: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:06.048758 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 476 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 476: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:06.048933 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 476 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.049036 20874 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 475.313, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I1014 19:27:06.049309 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.049396 22715 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Stepping down as leader of term 475
I1014 19:27:06.049437 22715 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:06.049710 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 476 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 476 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.049610 22715 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 475.313, Last appended by leader: 313, Current term: 475, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.049772 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 476 election: Election decided. Result: candidate lost.
I1014 19:27:06.049794 22715 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:06.049835 22715 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:06.049942 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:06.049959 22715 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 475 FOLLOWER]: Advancing to term 476
I1014 19:27:06.049989 22681 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 476 FOLLOWER]: Leader election lost for term 476. Reason: could not achieve majority
I1014 19:27:06.127352 22715 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 476 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:06.127449 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:06.127511 22715 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 476 FOLLOWER]: Advancing to term 477
I1014 19:27:06.128914 22715 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.129356 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 477 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.129542 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 477 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.130131 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 477 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:06.130159 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 477 candidate_status { last_received { term: 475 index: 313 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.130285 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 476 FOLLOWER]: Advancing to term 477
I1014 19:27:06.130326 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 476 FOLLOWER]: Advancing to term 477
I1014 19:27:06.130354 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.123s (vote granted)
I1014 19:27:06.130426 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (vote granted)
I1014 19:27:06.131848 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 477.
I1014 19:27:06.132202 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 477 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.132257 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 477 election: Election decided. Result: candidate won.
I1014 19:27:06.132452 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:06.132530 22715 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 FOLLOWER]: Leader election won for term 477
I1014 19:27:06.132556 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 477 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 477.
I1014 19:27:06.132602 22715 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:06.132920 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 477 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.132794 22715 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 475.313, Last appended by leader: 313, Current term: 477, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.133689 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.133880 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.223662 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 475 index: 313. Preceding OpId from leader: term: 477 index: 314. (index mismatch)
I1014 19:27:06.225212 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I1014 19:27:06.226390 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.226645 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.227905 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.228102 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.228337 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 477 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 475 index: 313. Preceding OpId from leader: term: 477 index: 314. (index mismatch)
I1014 19:27:06.228384 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.228591 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.228890 20874 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I1014 19:27:06.230533 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.230741 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.230923 22957 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.231127 22957 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.312825 20874 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 477 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.313112 20874 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.494216 22957 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 477 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:06.494288 22957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:06.494335 22957 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 477 FOLLOWER]: Advancing to term 478
I1014 19:27:06.495856 22957 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.496245 22957 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 478 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.496388 22957 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 478 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.496976 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 478 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:06.497179 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 478 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.497169 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 478 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.497354 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 478 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.497741 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 478 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 478 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.497838 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 478 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 478 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.497900 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 478 election: Election decided. Result: candidate lost.
I1014 19:27:06.498070 22957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:06.498126 22957 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Leader election lost for term 478. Reason: could not achieve majority
I1014 19:27:06.512872 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 477. Current term is 478. Ops: []
I1014 19:27:06.513398 22715 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 477.314, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I1014 19:27:06.513659 23019 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 LEADER]: Stepping down as leader of term 477
I1014 19:27:06.513706 23019 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:06.513871 23019 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 477.314, Last appended by leader: 314, Current term: 477, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.514045 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:06.514073 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:06.514163 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 477 FOLLOWER]: Advancing to term 478
I1014 19:27:06.587761 22681 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:06.587857 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:06.587913 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 477 FOLLOWER]: Advancing to term 478
I1014 19:27:06.589942 22681 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.590490 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 478 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.590662 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 478 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.591238 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 478 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.591238 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 478 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.591433 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 478: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:06.591418 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (vote granted)
I1014 19:27:06.593530 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 478 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 478.
I1014 19:27:06.593931 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 478 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.593999 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 478 election: Election decided. Result: candidate won.
I1014 19:27:06.594053 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 478 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 478: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:06.594190 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:06.594242 22681 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 FOLLOWER]: Leader election won for term 478
I1014 19:27:06.594532 22681 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:06.594734 22681 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 477.314, Last appended by leader: 314, Current term: 478, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.596068 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 478 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.596335 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.596954 23129 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:06.598389 23129 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:06.607461 22957 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:06.607570 22957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:06.607623 22957 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 478 FOLLOWER]: Advancing to term 479
I1014 19:27:06.609321 22957 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.609824 22957 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 479 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.609977 22957 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 479 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.610780 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 479 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.610949 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 478 FOLLOWER]: Advancing to term 479
I1014 19:27:06.611021 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:06.610952 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 479 candidate_status { last_received { term: 477 index: 314 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:06.611146 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 479 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.611663 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 479 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 479 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.612910 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 479.
I1014 19:27:06.613407 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 479 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.613488 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 479 election: Election decided. Result: candidate won.
I1014 19:27:06.613673 22957 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:06.613730 22957 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 FOLLOWER]: Leader election won for term 479
I1014 19:27:06.613801 22957 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:06.613981 22957 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 477.314, Last appended by leader: 314, Current term: 479, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.614873 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.615057 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.621130 23145 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:06.623054 23145 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:06.673807 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 478. Current term is 479. Ops: []
I1014 19:27:06.674465 23126 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I1014 19:27:06.674742 22681 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 LEADER]: Stepping down as leader of term 478
I1014 19:27:06.674795 22681 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:06.674991 22681 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 478.315, Last appended by leader: 315, Current term: 478, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.675191 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:06.675232 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:06.675341 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 478 FOLLOWER]: Advancing to term 479
I1014 19:27:06.724484 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 479 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 478 index: 315. Preceding OpId from leader: term: 479 index: 315. (term mismatch)
I1014 19:27:06.724537 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 314
I1014 19:27:06.724556 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 478.315
I1014 19:27:06.724591 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 479 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:06.725090 22394 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I1014 19:27:06.726394 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.726677 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.727413 22957 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.727627 22957 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.727991 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 477 index: 314. Preceding OpId from leader: term: 479 index: 315. (index mismatch)
I1014 19:27:06.727974 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.728147 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.728983 22394 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I1014 19:27:06.731712 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.731827 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.731988 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.732019 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:06.816860 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 479 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.817086 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.828311 22681 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 479 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:06.828384 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:06.828436 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 479 FOLLOWER]: Advancing to term 480
I1014 19:27:06.830026 22681 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.830433 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 480 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.830629 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 480 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.831182 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 480 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.831185 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 480 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.831351 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 480 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.831367 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 480 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.831779 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 480 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 480 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.831885 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 480 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 480 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.831950 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 480 election: Election decided. Result: candidate lost.
I1014 19:27:06.832135 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:06.832199 22681 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Leader election lost for term 480. Reason: could not achieve majority
I1014 19:27:06.845047 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 479. Current term is 480. Ops: []
I1014 19:27:06.845526 22394 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 479.315, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I1014 19:27:06.845743 23162 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 LEADER]: Stepping down as leader of term 479
I1014 19:27:06.845794 23162 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:06.845993 23162 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 479.315, Last appended by leader: 315, Current term: 479, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.846187 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:06.846216 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:06.846297 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 479 FOLLOWER]: Advancing to term 480
I1014 19:27:06.918740 23019 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:06.918836 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:06.918893 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 479 FOLLOWER]: Advancing to term 480
I1014 19:27:06.920657 23019 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.921090 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 480 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:06.921241 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 480 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.921859 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 480 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:06.922076 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 480: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:06.922088 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 480 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.922241 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (vote granted)
I1014 19:27:06.922435 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 480 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 480: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:06.924312 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 480 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 480.
I1014 19:27:06.924620 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 480 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.924665 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 480 election: Election decided. Result: candidate won.
I1014 19:27:06.924825 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:06.924897 23019 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 FOLLOWER]: Leader election won for term 480
I1014 19:27:06.924990 23019 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:06.925218 23019 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 479.315, Last appended by leader: 315, Current term: 480, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.926230 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 480 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.926477 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:06.959774 22681 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:06.959856 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.202s (starting election)
I1014 19:27:06.959910 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 480 FOLLOWER]: Advancing to term 481
I1014 19:27:06.962019 22681 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.962512 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 481 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.962657 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 481 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:06.963184 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 481 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:06.963186 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 481 candidate_status { last_received { term: 479 index: 315 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:06.963342 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 481 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.963325 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 480 FOLLOWER]: Advancing to term 481
I1014 19:27:06.963394 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:06.963690 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 481 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 481 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:06.965445 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 481.
I1014 19:27:06.965773 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 481 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:06.965831 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 481 election: Election decided. Result: candidate won.
I1014 19:27:06.965997 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.184s (election complete)
I1014 19:27:06.966050 22681 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 FOLLOWER]: Leader election won for term 481
I1014 19:27:06.966140 22681 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:06.966385 22681 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 479.315, Last appended by leader: 315, Current term: 481, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:06.967346 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:06.967526 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.018224 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 480. Current term is 481. Ops: []
I1014 19:27:07.018841 22715 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I1014 19:27:07.019106 22715 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 481.316 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 315
I1014 19:27:07.019178 23019 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 LEADER]: Stepping down as leader of term 480
I1014 19:27:07.019229 23019 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:07.019449 23019 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 480.316, Last appended by leader: 316, Current term: 480, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.019634 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:07.019677 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:07.019773 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 480 FOLLOWER]: Advancing to term 481
I1014 19:27:07.044958 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 479 index: 315. Preceding OpId from leader: term: 481 index: 316. (index mismatch)
I1014 19:27:07.045514 23126 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I1014 19:27:07.046795 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.047005 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.048611 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.048970 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.049001 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.049194 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.049190 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 481 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 480 index: 316. Preceding OpId from leader: term: 481 index: 316. (term mismatch)
I1014 19:27:07.049234 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 315
I1014 19:27:07.049257 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 480.316
I1014 19:27:07.049293 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 481 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:07.049935 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s
I1014 19:27:07.051250 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.051290 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.051589 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.051611 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.151994 23019 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 481 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:07.152079 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:07.152143 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 481 FOLLOWER]: Advancing to term 482
I1014 19:27:07.153767 23019 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.154207 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 482 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.154362 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 482 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.155002 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 482 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.155045 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 482 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.155162 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 482 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.155228 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 482 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.155565 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 482 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 482 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.155699 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 482 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 482 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.155751 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 482 election: Election decided. Result: candidate lost.
I1014 19:27:07.155946 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:07.156003 23019 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Leader election lost for term 482. Reason: could not achieve majority
I1014 19:27:07.171612 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 481 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.171938 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.172204 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 481. Current term is 482. Ops: []
I1014 19:27:07.172685 23126 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 481.316, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I1014 19:27:07.172849 23126 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Stepping down as leader of term 481
I1014 19:27:07.172891 23126 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:07.173053 23126 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 481.316, Last appended by leader: 316, Current term: 481, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.173167 23126 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:07.173190 23126 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:07.173267 23126 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 481 FOLLOWER]: Advancing to term 482
I1014 19:27:07.230211 23162 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:07.230303 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:07.230350 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 481 FOLLOWER]: Advancing to term 482
I1014 19:27:07.232471 23162 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.232990 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 482 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.233160 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 482 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.233768 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 482 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.233804 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 482 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:07.233937 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (vote granted)
I1014 19:27:07.234006 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 482: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:07.234446 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 482 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 482: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:07.235587 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 482 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 482.
I1014 19:27:07.235874 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 482 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.235932 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 482 election: Election decided. Result: candidate won.
I1014 19:27:07.236115 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:07.236225 23162 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 FOLLOWER]: Leader election won for term 482
I1014 19:27:07.236299 23162 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:07.236459 23162 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 481.316, Last appended by leader: 316, Current term: 482, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.237355 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 482 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.237565 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.298094 23019 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:07.298171 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.154s (starting election)
I1014 19:27:07.298221 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 482 FOLLOWER]: Advancing to term 483
I1014 19:27:07.300215 23019 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.300760 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 483 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.300916 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 483 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.301581 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 483 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.301584 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 483 candidate_status { last_received { term: 481 index: 316 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.301713 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 482 FOLLOWER]: Advancing to term 483
I1014 19:27:07.301765 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 483 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.301782 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:07.302145 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 483 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 483 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.303503 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 483.
I1014 19:27:07.303848 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 483 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.303902 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 483 election: Election decided. Result: candidate won.
I1014 19:27:07.304522 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.179s (election complete)
I1014 19:27:07.304584 23019 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 FOLLOWER]: Leader election won for term 483
I1014 19:27:07.304666 23019 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:07.304920 23019 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 481.316, Last appended by leader: 316, Current term: 483, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.306404 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.306587 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:27:07.326789 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.550s	user 0.002s	sys 0.000s
I1014 19:27:07.326843 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:07.327751 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:07.329020 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 482. Current term is 483. Ops: []
I1014 19:27:07.330927 23162 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I1014 19:27:07.331452 23414 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 LEADER]: Stepping down as leader of term 482
I1014 19:27:07.332348 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 482. Current term is 483. Ops: []
I1014 19:27:07.333364 23414 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:07.333750 23414 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 316, Committed index: 316, Last appended: 482.317, Last appended by leader: 317, Current term: 482, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.334637 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:07.334718 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:07.334890 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 482 FOLLOWER]: Advancing to term 483
I1014 19:27:07.337450 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 483 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 482 index: 317. Preceding OpId from leader: term: 483 index: 318. (index mismatch)
I1014 19:27:07.338330 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 481 index: 316. Preceding OpId from leader: term: 483 index: 318. (index mismatch)
I1014 19:27:07.339972 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I1014 19:27:07.340438 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 317, Last known committed idx: 316, Time since last communication: 0.000s
I1014 19:27:07.340608 23019 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 482.317 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 316
I1014 19:27:07.341683 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 316
I1014 19:27:07.341717 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 482.317
I1014 19:27:07.341763 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 483 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:07.343420 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.343699 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.349951 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.350281 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.350605 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:07.350600 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.350816 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.352571 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:07.355392 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.355639 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.356353 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.356586 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.356550 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 483 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.356720 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.445627 22394 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 483 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:07.445705 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:07.445758 22394 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 483 FOLLOWER]: Advancing to term 484
I1014 19:27:07.447296 22394 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.447703 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 484 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.447857 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 484 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.448302 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 484 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.448468 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 484 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.448422 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 484 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:07.448657 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 484 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.448845 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 484 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 484 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.448971 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 484 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 484 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.449059 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 484 election: Election decided. Result: candidate lost.
I1014 19:27:07.449218 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:07.449275 22394 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Leader election lost for term 484. Reason: could not achieve majority
I1014 19:27:07.468989 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 483. Current term is 484. Ops: []
I1014 19:27:07.469559 23019 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 483.318, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I1014 19:27:07.469820 22715 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Stepping down as leader of term 483
I1014 19:27:07.469867 22715 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:07.470070 22715 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 483.318, Last appended by leader: 318, Current term: 483, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.470252 22715 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:07.470281 22715 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:07.470368 22715 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 483 FOLLOWER]: Advancing to term 484
I1014 19:27:07.545639 23126 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:07.545722 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (starting election)
I1014 19:27:07.545774 23126 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 483 FOLLOWER]: Advancing to term 484
I1014 19:27:07.547946 23126 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.548465 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 484 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.548647 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 484 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.549190 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 484 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.549190 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 484 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:07.549383 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 484: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:07.549401 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (vote granted)
I1014 19:27:07.551023 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 484 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 484.
I1014 19:27:07.551326 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 484 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 484: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:07.551417 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 484 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.551466 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 484 election: Election decided. Result: candidate won.
I1014 19:27:07.551760 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:07.551817 23126 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 FOLLOWER]: Leader election won for term 484
I1014 19:27:07.551898 23126 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:07.552079 23126 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 483.318, Last appended by leader: 318, Current term: 484, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.553045 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 484 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.553169 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.556532 22394 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:07.556596 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.209s (starting election)
I1014 19:27:07.556650 22394 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 484 FOLLOWER]: Advancing to term 485
I1014 19:27:07.558636 22394 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.559044 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 485 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.559195 22394 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 485 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.559751 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 485 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.559737 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 485 candidate_status { last_received { term: 483 index: 318 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:07.559906 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 485 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.559906 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 484 FOLLOWER]: Advancing to term 485
I1014 19:27:07.559967 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:07.560276 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 485 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 485 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.561771 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 485 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 485.
I1014 19:27:07.562536 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 485 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.562583 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 485 election: Election decided. Result: candidate won.
I1014 19:27:07.562842 22394 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:07.562902 22394 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 FOLLOWER]: Leader election won for term 485
I1014 19:27:07.562976 22394 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:07.563144 22394 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 483.318, Last appended by leader: 318, Current term: 485, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.564097 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.564270 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.630213 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 484. Current term is 485. Ops: []
I1014 19:27:07.630726 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I1014 19:27:07.630971 22681 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 485.319 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 318
I1014 19:27:07.631063 22681 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 LEADER]: Stepping down as leader of term 484
I1014 19:27:07.631099 22681 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:07.631268 22681 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 484.319, Last appended by leader: 319, Current term: 484, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.631410 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:07.631448 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:07.631561 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 484 FOLLOWER]: Advancing to term 485
I1014 19:27:07.641901 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 484 index: 319. Preceding OpId from leader: term: 485 index: 319. (term mismatch)
I1014 19:27:07.641953 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 318
I1014 19:27:07.641973 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 484.319
I1014 19:27:07.642009 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:07.642596 23414 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I1014 19:27:07.644065 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.644316 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.644919 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.645112 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.645761 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.645937 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.646095 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 485 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 483 index: 318. Preceding OpId from leader: term: 485 index: 319. (index mismatch)
I1014 19:27:07.646581 22394 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I1014 19:27:07.647881 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.648121 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.647922 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.648175 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:07.748258 22715 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 485 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:07.748349 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:07.748421 22715 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 485 FOLLOWER]: Advancing to term 486
I1014 19:27:07.750762 22715 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.751312 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 486 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.752120 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 486 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.752946 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 486 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.753160 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 486 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.753722 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 486 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.753909 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 486 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.753914 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 486 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 486 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.754257 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 486 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 486 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.754323 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 486 election: Election decided. Result: candidate lost.
I1014 19:27:07.754623 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:07.754782 22715 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Leader election lost for term 486. Reason: could not achieve majority
I1014 19:27:07.767040 22394 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 485 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.767241 22394 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.770063 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 485. Current term is 486. Ops: []
I1014 19:27:07.771157 23162 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 485.319, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I1014 19:27:07.771371 23162 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Stepping down as leader of term 485
I1014 19:27:07.771400 23162 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:07.771570 23162 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 485.319, Last appended by leader: 319, Current term: 485, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.771700 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:07.771731 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:07.771818 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 485 FOLLOWER]: Advancing to term 486
I1014 19:27:07.839663 23126 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:07.839756 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:07.839808 23126 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 485 FOLLOWER]: Advancing to term 486
I1014 19:27:07.841346 23126 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.841761 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 486 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.841918 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 486 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:07.842532 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 486 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:07.842542 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 486 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.842667 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:07.842697 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 486: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:07.843091 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 486 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 486: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:07.844136 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 486 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 486.
I1014 19:27:07.844434 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 486 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.844504 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 486 election: Election decided. Result: candidate won.
I1014 19:27:07.844677 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:07.844743 23126 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 FOLLOWER]: Leader election won for term 486
I1014 19:27:07.844820 23126 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:07.845008 23126 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 485.319, Last appended by leader: 319, Current term: 486, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.845772 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 486 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.845984 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.894533 22715 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:07.894618 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:07.894675 22715 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 486 FOLLOWER]: Advancing to term 487
I1014 19:27:07.896872 22715 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.897344 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 487 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:07.897531 22715 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 487 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.898068 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 487 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:07.898106 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 487 candidate_status { last_received { term: 485 index: 319 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:07.898238 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 487 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.898247 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 486 FOLLOWER]: Advancing to term 487
I1014 19:27:07.898313 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:27:07.900380 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 487.
I1014 19:27:07.901034 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 487 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:07.901099 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 487 election: Election decided. Result: candidate won.
I1014 19:27:07.901299 22715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.200s (election complete)
I1014 19:27:07.901350 22715 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 FOLLOWER]: Leader election won for term 487
I1014 19:27:07.901348 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 487 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 487 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:07.901446 22715 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:07.901695 22715 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 485.319, Last appended by leader: 319, Current term: 487, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.903216 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:07.903362 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:07.961307 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 486. Current term is 487. Ops: []
I1014 19:27:07.961828 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I1014 19:27:07.962023 22681 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 487.320 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 319
I1014 19:27:07.962079 23126 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 LEADER]: Stepping down as leader of term 486
I1014 19:27:07.962127 23126 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:07.962302 23126 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 486.320, Last appended by leader: 320, Current term: 486, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:07.962489 23126 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:07.962551 23126 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:07.962649 23126 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 486 FOLLOWER]: Advancing to term 487
W1014 19:27:08.001130 23129 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.403s	user 0.002s	sys 0.000s
I1014 19:27:08.001175 23129 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:08.001987 23129 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:08.003044 23129 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 478 vs 487: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.003316 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.003541 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.003686 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.003808 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.003988 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 487 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.009111 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 485 index: 319. Preceding OpId from leader: term: 487 index: 320. (index mismatch)
I1014 19:27:08.009726 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I1014 19:27:08.009924 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 487 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 486 index: 320. Preceding OpId from leader: term: 487 index: 320. (term mismatch)
I1014 19:27:08.009979 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 319
I1014 19:27:08.010002 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 486.320
I1014 19:27:08.010043 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 487 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:08.010560 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I1014 19:27:08.010989 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.011181 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.011654 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.011946 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.012061 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.012249 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.012892 22715 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.013131 22715 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.013207 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.013439 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.014003 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 487 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.014189 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.114382 23126 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 487 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:08.114504 23126 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:08.114567 23126 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 487 FOLLOWER]: Advancing to term 488
I1014 19:27:08.116820 23126 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.117350 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 488 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.118386 23126 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 488 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.118484 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 488 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.118698 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 488 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.119061 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 488 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.119199 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 488 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 488 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.119268 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 488 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.119657 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 488 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 488 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.119716 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 488 election: Election decided. Result: candidate lost.
I1014 19:27:08.119930 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:08.120003 22681 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Leader election lost for term 488. Reason: could not achieve majority
I1014 19:27:08.128515 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 487. Current term is 488. Ops: []
I1014 19:27:08.128988 23019 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 487.320, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I1014 19:27:08.129276 23673 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Stepping down as leader of term 487
I1014 19:27:08.129323 23673 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:08.129515 23673 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 487.320, Last appended by leader: 320, Current term: 487, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.129716 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:08.129755 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:08.129864 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 487 FOLLOWER]: Advancing to term 488
I1014 19:27:08.156471 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:08.157379 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:08.158612 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 483 vs 488: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.159129 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.159538 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.159979 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.160410 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.160836 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.161259 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.161720 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.162207 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.162657 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.163112 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.163544 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 488 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.190735 23162 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:08.190819 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:08.190871 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 487 FOLLOWER]: Advancing to term 488
I1014 19:27:08.192910 23162 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.193413 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 488 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.193593 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 488 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.194506 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 488 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.194677 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:08.194721 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 488 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:08.194903 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 488: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:08.195478 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 488 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 488: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:08.197702 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 488 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 488.
I1014 19:27:08.198689 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 488 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.198746 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 488 election: Election decided. Result: candidate won.
I1014 19:27:08.199100 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:08.199198 23162 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 FOLLOWER]: Leader election won for term 488
I1014 19:27:08.199326 23162 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:08.199628 23162 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 487.320, Last appended by leader: 320, Current term: 488, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.201398 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 488 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.201619 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.221343 22681 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:08.221427 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:08.221493 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 488 FOLLOWER]: Advancing to term 489
I1014 19:27:08.223583 22681 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.224064 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 489 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.224228 22681 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 489 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.225978 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 489 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.225977 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 489 candidate_status { last_received { term: 487 index: 320 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.226138 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 488 FOLLOWER]: Advancing to term 489
I1014 19:27:08.226169 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 489 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.226198 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:08.226577 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 489 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 489 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.228210 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 489.
I1014 19:27:08.228498 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 489 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.228581 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 489 election: Election decided. Result: candidate won.
I1014 19:27:08.228750 22681 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.196s (election complete)
I1014 19:27:08.228808 22681 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 FOLLOWER]: Leader election won for term 489
I1014 19:27:08.228912 22681 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:08.229166 22681 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 487.320, Last appended by leader: 320, Current term: 489, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.230554 23126 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.230803 23126 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.230960 23129 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:08.232743 23129 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:08.313201 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 488. Current term is 489. Ops: []
I1014 19:27:08.314147 23414 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I1014 19:27:08.314364 23414 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 489.321 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 320
I1014 19:27:08.314466 23162 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 LEADER]: Stepping down as leader of term 488
I1014 19:27:08.314558 23162 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:08.314769 23162 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 488.321, Last appended by leader: 321, Current term: 488, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.314970 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:08.315038 23162 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:08.315174 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 488 FOLLOWER]: Advancing to term 489
I1014 19:27:08.318246 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 487 index: 320. Preceding OpId from leader: term: 489 index: 321. (index mismatch)
I1014 19:27:08.318722 23126 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I1014 19:27:08.319767 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.319923 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.320947 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.321202 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.321499 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.321749 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.321840 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 489 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 488 index: 321. Preceding OpId from leader: term: 489 index: 321. (term mismatch)
I1014 19:27:08.321899 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 320
I1014 19:27:08.321928 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 488.321
I1014 19:27:08.321979 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 489 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:08.322458 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I1014 19:27:08.323803 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.323851 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.324204 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.324206 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:08.390996 23145 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.768s	user 0.002s	sys 0.000s
I1014 19:27:08.391057 23145 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:08.391955 23145 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:08.398097 23145 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 479 vs 489: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.398490 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.398773 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.399027 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.400642 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.401396 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.403782 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.404048 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.424623 23162 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 489 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:08.424712 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:08.424779 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 489 FOLLOWER]: Advancing to term 490
I1014 19:27:08.426831 23162 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.427377 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 490 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.427573 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 490 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.428158 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 490 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:08.428160 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 490 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.428349 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 490 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.428357 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 490 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.428756 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 490 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 490 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.428952 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 490 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 490 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.429019 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 490 election: Election decided. Result: candidate lost.
I1014 19:27:08.429352 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:08.429486 23162 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Leader election lost for term 490. Reason: could not achieve majority
I1014 19:27:08.429291 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 489 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.429733 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.430268 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 489. Current term is 490. Ops: []
I1014 19:27:08.430665 23129 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:08.430686 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 489.321, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I1014 19:27:08.430918 23782 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Stepping down as leader of term 489
I1014 19:27:08.431010 23782 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:08.431262 23782 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 489.321, Last appended by leader: 321, Current term: 489, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.431462 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:08.431499 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:08.431646 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 489 FOLLOWER]: Advancing to term 490
I1014 19:27:08.431687 23129 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:08.440254 23129 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 489 vs 490: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 490 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.440593 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 490 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.440807 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 490 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.520259 23673 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:08.520351 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:27:08.520432 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 489 FOLLOWER]: Advancing to term 490
I1014 19:27:08.522680 23673 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.523200 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 490 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.523355 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 490 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.524143 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 490 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:08.524319 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (vote granted)
I1014 19:27:08.526358 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 490 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 490.
I1014 19:27:08.526914 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 490 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.526988 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 490 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.527052 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 490 election: Election decided. Result: candidate won.
I1014 19:27:08.527128 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 490: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:08.527298 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:08.527580 23673 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 FOLLOWER]: Leader election won for term 490
I1014 19:27:08.527611 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 490 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 490: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:08.527833 23673 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:08.528177 23673 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 489.321, Last appended by leader: 321, Current term: 490, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.530009 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 490 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.530171 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.530369 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:08.532126 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:08.578958 23162 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:08.579088 23162 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.208s (starting election)
I1014 19:27:08.579200 23162 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 490 FOLLOWER]: Advancing to term 491
I1014 19:27:08.581409 23162 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.582048 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 491 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.582708 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 491 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:08.582877 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 490 FOLLOWER]: Advancing to term 491
I1014 19:27:08.582954 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (vote granted)
I1014 19:27:08.585038 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 491 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 491.
I1014 19:27:08.585644 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 491 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.585743 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 491 election: Election decided. Result: candidate won.
I1014 19:27:08.585992 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.190s (election complete)
I1014 19:27:08.586074 23414 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 FOLLOWER]: Leader election won for term 491
I1014 19:27:08.587059 23162 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 491 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.587739 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 491 candidate_status { last_received { term: 489 index: 321 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.587935 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 491 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.588304 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 491 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 491 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.590680 23414 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:08.590942 23414 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 489.321, Last appended by leader: 321, Current term: 491, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.592300 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.592538 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.592739 23145 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:08.594683 23145 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:08.623399 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 490. Current term is 491. Ops: []
I1014 19:27:08.623883 23019 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I1014 19:27:08.624105 23019 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 491.322 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 321
I1014 19:27:08.624163 23673 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 LEADER]: Stepping down as leader of term 490
I1014 19:27:08.624202 23673 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:08.624370 23673 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 490.322, Last appended by leader: 322, Current term: 490, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.624604 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:08.624646 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:08.624735 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 490 FOLLOWER]: Advancing to term 491
I1014 19:27:08.670295 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 491 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 489 index: 321. Preceding OpId from leader: term: 491 index: 322. (index mismatch)
I1014 19:27:08.670862 23162 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I1014 19:27:08.672045 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.672332 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.673311 23933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.673576 23933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.673877 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 491 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 490 index: 322. Preceding OpId from leader: term: 491 index: 322. (term mismatch)
I1014 19:27:08.673920 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 321
I1014 19:27:08.673938 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 490.322
I1014 19:27:08.673797 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.673967 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 491 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:08.673974 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.674559 23162 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I1014 19:27:08.675698 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.675971 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.675933 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.676189 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:08.774842 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 491 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:08.774919 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:08.774966 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 491 FOLLOWER]: Advancing to term 492
I1014 19:27:08.775894 23673 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 491 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:08.775961 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:08.776002 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 491 FOLLOWER]: Advancing to term 492
I1014 19:27:08.777074 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.777592 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 492 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.777748 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 492 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.778143 23673 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.778214 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 492 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.778427 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 492 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.778585 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 492 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:08.778723 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 492 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.778820 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 492 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 492 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.779353 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 492 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.779541 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 492: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:08.779800 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 492 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.779850 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 492 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:08.779954 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 492 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.779963 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 492: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:08.780236 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 492 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 492: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:08.780282 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 492 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 492 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:08.780306 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 492 election: Election decided. Result: candidate lost.
I1014 19:27:08.780489 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:08.780542 23782 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Leader election lost for term 492. Reason: could not achieve majority
I1014 19:27:08.780830 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 491. Current term is 492. Ops: []
I1014 19:27:08.781311 23933 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 491.322, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I1014 19:27:08.781309 23162 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 491 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.781612 23414 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Stepping down as leader of term 491
I1014 19:27:08.781657 23162 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.781656 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 492 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 492: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:08.781711 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 492 election: Election decided. Result: candidate lost.
I1014 19:27:08.781709 23414 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:08.781860 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:08.781910 23673 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Leader election lost for term 492. Reason: could not achieve majority
I1014 19:27:08.781973 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 491. Current term is 492. Ops: []
I1014 19:27:08.781896 23414 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 491.322, Last appended by leader: 322, Current term: 491, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.782088 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:08.782157 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:08.782276 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 491 FOLLOWER]: Advancing to term 492
I1014 19:27:08.890131 23145 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:08.891060 23145 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:08.892096 23145 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 491 vs 492: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 492 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:08.892359 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 492 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.895728 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:08.895804 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:08.895859 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 492 FOLLOWER]: Advancing to term 493
W1014 19:27:08.895956 23145 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 492 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.897851 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.898324 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 493 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.898504 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 493 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.899718 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 493 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:08.899883 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 492 FOLLOWER]: Advancing to term 493
I1014 19:27:08.899966 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:08.901859 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 493 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 493.
I1014 19:27:08.902209 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 493 candidate_status { last_received { term: 491 index: 322 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:08.902369 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 492 FOLLOWER]: Advancing to term 493
I1014 19:27:08.902457 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.153s (vote granted)
I1014 19:27:08.904330 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 493.
I1014 19:27:08.904835 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 493 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:08.904981 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 493 election: Election decided. Result: candidate won.
I1014 19:27:08.905201 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 493 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:08.905248 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.206s (election complete)
I1014 19:27:08.905328 23782 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 FOLLOWER]: Leader election won for term 493
I1014 19:27:08.905427 23782 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:08.905652 23782 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 491.322, Last appended by leader: 322, Current term: 493, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:08.906646 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:08.906908 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:08.907073 23129 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:08.908839 23129 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:09.014612 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 493 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 491 index: 322. Preceding OpId from leader: term: 493 index: 323. (index mismatch)
I1014 19:27:09.015167 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I1014 19:27:09.016327 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.016654 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.017339 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.017601 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.018005 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 491 index: 322. Preceding OpId from leader: term: 493 index: 323. (index mismatch)
I1014 19:27:09.017917 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.018110 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.018422 24150 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I1014 19:27:09.019801 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.019964 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.019763 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.020031 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.107841 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.108038 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.136198 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:09.137100 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:09.138152 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 490 vs 493: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 493 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.196112 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 493 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:09.196207 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:09.196280 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 493 FOLLOWER]: Advancing to term 494
I1014 19:27:09.198387 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.198880 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 494 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.199036 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 494 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.200109 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 494 candidate_status { last_received { term: 493 index: 323 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.200294 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 494 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.200461 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 494 candidate_status { last_received { term: 493 index: 323 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.200667 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 494 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.202020 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 493. Current term is 494. Ops: []
I1014 19:27:09.202364 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 494 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 494 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.202796 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 493.323, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I1014 19:27:09.203021 23782 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 LEADER]: Stepping down as leader of term 493
I1014 19:27:09.203059 23782 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:09.203260 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 494 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 494 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.203333 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 494 election: Election decided. Result: candidate lost.
I1014 19:27:09.203222 23782 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 493.323, Last appended by leader: 323, Current term: 493, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.203393 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:09.203430 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:09.203586 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 493 FOLLOWER]: Advancing to term 494
I1014 19:27:09.203734 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:09.203899 23414 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Leader election lost for term 494. Reason: could not achieve majority
I1014 19:27:09.215605 23673 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:09.215678 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:09.215728 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 493 FOLLOWER]: Advancing to term 494
I1014 19:27:09.217180 23673 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.217662 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 494 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.217823 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 494 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:09.218389 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 494 candidate_status { last_received { term: 493 index: 323 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.218474 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 494 candidate_status { last_received { term: 493 index: 323 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:09.218587 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (vote granted)
I1014 19:27:09.218672 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 494: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:09.219056 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 494 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 494: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:09.220654 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 494 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 494.
I1014 19:27:09.221465 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 494 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.221534 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 494 election: Election decided. Result: candidate won.
I1014 19:27:09.221734 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:09.221788 23673 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 FOLLOWER]: Leader election won for term 494
I1014 19:27:09.221860 23673 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:09.222046 23673 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 493.323, Last appended by leader: 323, Current term: 494, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.223050 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.223255 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.223501 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:09.225509 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:09.309900 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 494 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 493 index: 323. Preceding OpId from leader: term: 494 index: 324. (index mismatch)
I1014 19:27:09.310381 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I1014 19:27:09.311492 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.311720 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.312880 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.313061 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.313794 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 493 index: 323. Preceding OpId from leader: term: 494 index: 324. (index mismatch)
I1014 19:27:09.313752 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.313930 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.314337 23673 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I1014 19:27:09.316288 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.316323 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.316597 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.316702 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.402043 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 494 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.402248 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.414382 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 494 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:09.414477 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:09.414528 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 494 FOLLOWER]: Advancing to term 495
I1014 19:27:09.416668 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.417153 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 495 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:09.417304 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 495 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.417861 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 495 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:09.418046 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 495 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.418464 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 495 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 495 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.421084 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 495 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.421267 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 495 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.421653 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 495 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 495 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.421723 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 495 election: Election decided. Result: candidate lost.
I1014 19:27:09.422089 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:09.422158 23782 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Leader election lost for term 495. Reason: could not achieve majority
I1014 19:27:09.431314 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 494. Current term is 495. Ops: []
I1014 19:27:09.431972 23673 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 494.324, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I1014 19:27:09.432262 23019 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 LEADER]: Stepping down as leader of term 494
I1014 19:27:09.432314 23019 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:09.432500 23019 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 494.324, Last appended by leader: 324, Current term: 494, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.432732 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:09.432771 23019 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:09.432898 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 494 FOLLOWER]: Advancing to term 495
I1014 19:27:09.503890 24314 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:09.503980 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:09.504068 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 494 FOLLOWER]: Advancing to term 495
I1014 19:27:09.507527 24314 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.508414 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 495 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.508666 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 495 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.509156 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 495 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.509359 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 495: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:09.509323 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 495 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.509490 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (vote granted)
I1014 19:27:09.509826 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 495 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 495: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:09.511862 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 495 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 495.
I1014 19:27:09.512253 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 495 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.512321 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 495 election: Election decided. Result: candidate won.
I1014 19:27:09.512723 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:09.512838 24314 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 FOLLOWER]: Leader election won for term 495
I1014 19:27:09.513072 24314 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:09.513319 24314 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 494.324, Last appended by leader: 324, Current term: 495, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.514752 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 495 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.514955 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.520973 24351 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:09.528976 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:09.529055 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:09.529112 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 495 FOLLOWER]: Advancing to term 496
I1014 19:27:09.531252 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.531738 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 496 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:09.531904 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 496 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.532508 24351 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:09.532752 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 496 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.532707 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 496 candidate_status { last_received { term: 494 index: 324 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:09.532924 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 495 FOLLOWER]: Advancing to term 496
I1014 19:27:09.532994 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (vote granted)
I1014 19:27:09.533017 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 496 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.534534 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 496.
I1014 19:27:09.534839 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 496 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 496 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.535181 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 496 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.535228 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 496 election: Election decided. Result: candidate won.
I1014 19:27:09.535406 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:09.535495 23782 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 FOLLOWER]: Leader election won for term 496
I1014 19:27:09.535604 23782 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:09.535784 23782 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 494.324, Last appended by leader: 324, Current term: 496, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.536734 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.536859 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.602983 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 495. Current term is 496. Ops: []
I1014 19:27:09.603838 23414 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I1014 19:27:09.604050 23414 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 496.325 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 324
I1014 19:27:09.604161 23414 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 LEADER]: Stepping down as leader of term 495
I1014 19:27:09.604214 23414 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:09.604434 23414 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 324, Committed index: 324, Last appended: 495.325, Last appended by leader: 325, Current term: 495, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.604636 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:09.604699 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:09.604874 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 495 FOLLOWER]: Advancing to term 496
I1014 19:27:09.615485 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 496 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 495 index: 325. Preceding OpId from leader: term: 496 index: 325. (term mismatch)
I1014 19:27:09.615586 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 324
I1014 19:27:09.615628 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 495.325
I1014 19:27:09.615696 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 496 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:09.616355 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I1014 19:27:09.617837 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.618084 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.618881 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.619086 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.619911 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 494 index: 324. Preceding OpId from leader: term: 496 index: 325. (index mismatch)
I1014 19:27:09.620477 23782 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 325, Last known committed idx: 324, Time since last communication: 0.000s
I1014 19:27:09.621865 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.622074 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.621902 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.622135 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.624334 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.624554 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.720475 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 496 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.720762 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.725416 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 496 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:09.725517 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:09.725569 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 496 FOLLOWER]: Advancing to term 497
I1014 19:27:09.726969 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.727221 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 496. Current term is 497. Ops: []
I1014 19:27:09.727427 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 497 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.727594 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 497 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.728149 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 497 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.728353 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 497 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.728658 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 497 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.728855 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 497 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.729182 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 497 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 497 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.729110 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 496.325, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I1014 19:27:09.729368 22681 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Stepping down as leader of term 496
I1014 19:27:09.729431 22681 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:09.729657 22681 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 496.325, Last appended by leader: 325, Current term: 496, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.729851 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:09.729918 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:09.730077 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 FOLLOWER]: Advancing to term 497
I1014 19:27:09.732491 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 497 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 496 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 497 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.732561 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 497 election: Election decided. Result: candidate lost.
I1014 19:27:09.732751 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:09.732810 23414 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Leader election lost for term 497. Reason: could not achieve majority
I1014 19:27:09.822937 23019 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:09.823022 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:09.823087 23019 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 496 FOLLOWER]: Advancing to term 497
I1014 19:27:09.824561 23019 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.824925 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 497 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.825057 23019 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 497 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:09.825693 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 497 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:09.825716 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 497 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.825877 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 497: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:09.825886 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (vote granted)
I1014 19:27:09.826313 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 497 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 497: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:09.827934 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 497 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 497.
I1014 19:27:09.828305 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 497 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.828353 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 497 election: Election decided. Result: candidate won.
I1014 19:27:09.829062 23019 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:09.829125 23019 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 FOLLOWER]: Leader election won for term 497
I1014 19:27:09.829206 23019 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:09.829391 23019 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 496.325, Last appended by leader: 325, Current term: 497, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.830176 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 497 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.830354 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.872376 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:09.872453 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:09.872534 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 497 FOLLOWER]: Advancing to term 498
I1014 19:27:09.874617 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.875103 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 498 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.875274 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 498 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:09.875804 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 498 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:09.875983 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 497 FOLLOWER]: Advancing to term 498
I1014 19:27:09.875984 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 498 candidate_status { last_received { term: 496 index: 325 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:09.876075 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:09.876163 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 498 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.876533 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 498 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 498 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:09.878204 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 498.
I1014 19:27:09.878559 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 498 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:09.878612 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 498 election: Election decided. Result: candidate won.
I1014 19:27:09.879251 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.154s (election complete)
I1014 19:27:09.879343 23414 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 FOLLOWER]: Leader election won for term 498
I1014 19:27:09.880439 23414 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:09.880694 23414 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 496.325, Last appended by leader: 325, Current term: 498, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.881935 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.882100 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.927815 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 497. Current term is 498. Ops: []
I1014 19:27:09.928375 23673 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I1014 19:27:09.928602 23673 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 498.326 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 325
I1014 19:27:09.928699 23673 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 LEADER]: Stepping down as leader of term 497
I1014 19:27:09.928743 23673 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:09.928930 23673 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 497.326, Last appended by leader: 326, Current term: 497, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:09.929114 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:09.929172 23673 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:09.929294 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 497 FOLLOWER]: Advancing to term 498
I1014 19:27:09.972383 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 498 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 497 index: 326. Preceding OpId from leader: term: 498 index: 326. (term mismatch)
I1014 19:27:09.972434 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 325
I1014 19:27:09.972455 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 497.326
I1014 19:27:09.972515 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 498 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:09.974009 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 496 index: 325. Preceding OpId from leader: term: 498 index: 326. (index mismatch)
I1014 19:27:09.974054 23414 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I1014 19:27:09.975668 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.975805 23414 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I1014 19:27:09.975875 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.977149 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.977334 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.977316 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.977548 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.977828 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.978065 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:09.981240 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.981415 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:09.982590 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 498 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:09.982812 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.078831 23673 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 498 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:10.078909 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:10.078958 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 498 FOLLOWER]: Advancing to term 499
I1014 19:27:10.081162 23673 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.081655 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 499 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.081817 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 499 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.082340 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 499 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:10.082572 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 499 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.082633 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 499 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.082811 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 499 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.082900 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 499 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 499 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.083176 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 499 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 499 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.083217 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 499 election: Election decided. Result: candidate lost.
I1014 19:27:10.083376 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:10.083487 23673 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Leader election lost for term 499. Reason: could not achieve majority
I1014 19:27:10.095966 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 498. Current term is 499. Ops: []
I1014 19:27:10.096679 23414 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 498.326, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I1014 19:27:10.096900 24314 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Stepping down as leader of term 498
I1014 19:27:10.096946 24314 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:10.097111 24314 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 498.326, Last appended by leader: 326, Current term: 498, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.097230 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:10.097255 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:10.097370 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 498 FOLLOWER]: Advancing to term 499
I1014 19:27:10.143373 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:10.144327 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:10.145385 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 494 vs 499: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 499 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.145686 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 499 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.146370 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 499 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.146698 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 499 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.169704 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:10.169803 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:10.169875 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 498 FOLLOWER]: Advancing to term 499
I1014 19:27:10.171540 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.172006 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 499 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.172222 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 499 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.172705 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 499 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.172873 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (vote granted)
I1014 19:27:10.173319 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 499 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:10.173540 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 499: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:10.173975 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 499 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 499: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:10.174978 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 499 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 499.
I1014 19:27:10.175782 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 499 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.175879 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 499 election: Election decided. Result: candidate won.
I1014 19:27:10.176931 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:10.177009 23782 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 FOLLOWER]: Leader election won for term 499
I1014 19:27:10.177127 23782 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:10.177356 23782 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 498.326, Last appended by leader: 326, Current term: 499, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.178292 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 499 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.178536 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.206789 23673 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:10.206866 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.211s (starting election)
I1014 19:27:10.206918 23673 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 499 FOLLOWER]: Advancing to term 500
I1014 19:27:10.208732 23673 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.209228 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 500 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.209391 23673 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 500 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.213639 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 500 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:10.213826 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 500 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.214156 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 500 candidate_status { last_received { term: 498 index: 326 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.214300 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 499 FOLLOWER]: Advancing to term 500
I1014 19:27:10.214360 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:10.214619 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 500 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 500 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.216390 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 500.
I1014 19:27:10.216794 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 500 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.216845 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 500 election: Election decided. Result: candidate won.
I1014 19:27:10.217464 23673 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.181s (election complete)
I1014 19:27:10.217530 23673 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 FOLLOWER]: Leader election won for term 500
I1014 19:27:10.217627 23673 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:10.217820 23673 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 498.326, Last appended by leader: 326, Current term: 500, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.218888 23019 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.219072 23019 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.219250 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:10.220734 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:10.287108 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 499. Current term is 500. Ops: []
I1014 19:27:10.287621 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I1014 19:27:10.287854 22681 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 500.327 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 326
I1014 19:27:10.287945 23782 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 LEADER]: Stepping down as leader of term 499
I1014 19:27:10.287997 23782 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:10.288197 23782 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 499.327, Last appended by leader: 327, Current term: 499, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.288347 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:10.288401 23782 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:10.288530 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 499 FOLLOWER]: Advancing to term 500
I1014 19:27:10.312839 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 500 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 499 index: 327. Preceding OpId from leader: term: 500 index: 327. (term mismatch)
I1014 19:27:10.312922 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 326
I1014 19:27:10.312952 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 499.327
I1014 19:27:10.312995 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 500 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:10.313660 23019 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I1014 19:27:10.314800 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.315058 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.316047 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 498 index: 326. Preceding OpId from leader: term: 500 index: 327. (index mismatch)
I1014 19:27:10.316313 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.316548 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.316949 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.317137 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.317281 23673 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s
I1014 19:27:10.318751 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.318935 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.318898 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.319077 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:10.391692 23129 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.483s	user 0.002s	sys 0.000s
I1014 19:27:10.391748 23129 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:10.392505 23129 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:10.393513 23129 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 493 vs 500: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.394574 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.394850 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.395148 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.395387 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.395694 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.395946 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.399569 23673 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 500 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.399822 23673 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.417484 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 500 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:10.417608 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:10.417698 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 500 FOLLOWER]: Advancing to term 501
I1014 19:27:10.419829 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.420321 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 501 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.420997 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 501 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.421118 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 501 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.421177 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 501 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.421869 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 501 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 501 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.421862 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 501 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:10.422068 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 501 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.422425 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 501 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 501 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.422497 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 501 election: Election decided. Result: candidate lost.
I1014 19:27:10.422837 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:10.422905 23782 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Leader election lost for term 501. Reason: could not achieve majority
I1014 19:27:10.427474 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 500. Current term is 501. Ops: []
I1014 19:27:10.428131 23019 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 500.327, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I1014 19:27:10.428433 24607 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Stepping down as leader of term 500
I1014 19:27:10.428527 24607 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:10.428755 24607 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 500.327, Last appended by leader: 327, Current term: 500, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.428989 24607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:10.429033 24607 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:10.429159 24607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 500 FOLLOWER]: Advancing to term 501
I1014 19:27:10.501060 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:10.501139 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:10.501196 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 500 FOLLOWER]: Advancing to term 501
I1014 19:27:10.502988 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.503427 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 501 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.503568 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 501 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.504155 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 501 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:10.504261 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 501 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:10.504364 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 501: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:10.504420 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (vote granted)
I1014 19:27:10.504828 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 501 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 501: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:10.506566 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 501 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 501.
I1014 19:27:10.507019 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 501 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.507072 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 501 election: Election decided. Result: candidate won.
I1014 19:27:10.507320 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:10.507385 23414 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 FOLLOWER]: Leader election won for term 501
I1014 19:27:10.507495 23414 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:10.507668 23414 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 500.327, Last appended by leader: 327, Current term: 501, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.508536 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 501 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.508808 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.545567 23782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:10.545662 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.224s (starting election)
I1014 19:27:10.545737 23782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 501 FOLLOWER]: Advancing to term 502
I1014 19:27:10.547700 23782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.548174 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 502 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.548355 23782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 502 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.548936 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 502 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.549121 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 502 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.549684 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 502 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 502 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.559720 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 502 candidate_status { last_received { term: 500 index: 327 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:10.559883 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 501 FOLLOWER]: Advancing to term 502
I1014 19:27:10.559958 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (vote granted)
I1014 19:27:10.561873 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 502 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 502.
I1014 19:27:10.562719 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 502 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.562793 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 502 election: Election decided. Result: candidate won.
W1014 19:27:10.563853 24351 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.031s	user 0.001s	sys 0.000s
I1014 19:27:10.563894 24351 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:10.564815 24351 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:10.565330 23782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.166s (election complete)
I1014 19:27:10.565486 23782 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 FOLLOWER]: Leader election won for term 502
I1014 19:27:10.565850 23782 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:10.566293 23782 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 500.327, Last appended by leader: 327, Current term: 502, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.568195 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 502 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 501. Current term is 502. Ops: []
I1014 19:27:10.568924 24314 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I1014 19:27:10.569128 24314 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 LEADER]: Stepping down as leader of term 501
I1014 19:27:10.569165 24314 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:10.569325 24314 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 501.329, Last appended by leader: 329, Current term: 501, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.569523 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:10.569563 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:10.569648 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 501 FOLLOWER]: Advancing to term 502
I1014 19:27:10.571169 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 501. Current term is 502. Ops: []
I1014 19:27:10.571599 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.571790 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.572130 23129 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:10.573943 23129 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:10.648267 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 502 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 500 index: 327. Preceding OpId from leader: term: 502 index: 328. (index mismatch)
I1014 19:27:10.649160 23782 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I1014 19:27:10.650257 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.650542 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.651535 23782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.651720 23782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.652024 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 501 index: 329. Preceding OpId from leader: term: 502 index: 328. (term mismatch)
I1014 19:27:10.652065 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 327
I1014 19:27:10.652083 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 501.328
I1014 19:27:10.652113 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:10.652151 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 501.329
I1014 19:27:10.652679 24351 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 495 vs 502: Aborted: Transaction aborted by new leader
W1014 19:27:10.652964 24351 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 502 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.652907 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.653111 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:10.653205 24351 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 502 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.653409 24351 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 502 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:10.653652 24351 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 502 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.654546 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I1014 19:27:10.656492 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.656663 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.656441 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.656710 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.736374 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 502 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.736624 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.846153 24607 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 502 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:10.846235 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:10.846307 24607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 502 FOLLOWER]: Advancing to term 503
I1014 19:27:10.848616 24607 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.849138 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 503 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.849503 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 503 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:10.849869 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 503 candidate_status { last_received { term: 502 index: 328 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:10.850064 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 503 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.850082 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 503 candidate_status { last_received { term: 502 index: 328 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:10.850219 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 503 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.850383 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 503 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 503 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.850572 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 503 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 503 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:10.850637 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 503 election: Election decided. Result: candidate lost.
I1014 19:27:10.850955 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:10.851023 24607 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Leader election lost for term 503. Reason: could not achieve majority
I1014 19:27:10.857755 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 502. Current term is 503. Ops: []
I1014 19:27:10.858314 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 502.328, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I1014 19:27:10.858574 24617 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Stepping down as leader of term 502
I1014 19:27:10.858619 24617 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:10.858788 24617 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 502.328, Last appended by leader: 328, Current term: 502, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.858968 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:10.858999 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:10.859102 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 502 FOLLOWER]: Advancing to term 503
I1014 19:27:10.924290 24314 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:10.924383 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:10.924434 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 502 FOLLOWER]: Advancing to term 503
I1014 19:27:10.926532 24314 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.926997 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 503 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.927152 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 503 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:10.927798 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 503 candidate_status { last_received { term: 502 index: 328 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:10.927976 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (vote granted)
I1014 19:27:10.927950 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 503 candidate_status { last_received { term: 502 index: 328 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:10.928184 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 503: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:10.928835 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 503 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 503: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:10.929952 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 503 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 503.
I1014 19:27:10.930330 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 503 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:10.930385 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 503 election: Election decided. Result: candidate won.
I1014 19:27:10.931234 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:10.931293 24314 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 FOLLOWER]: Leader election won for term 503
I1014 19:27:10.931390 24314 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:10.931749 24314 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 502.328, Last appended by leader: 328, Current term: 503, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:10.937714 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.937902 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.938092 24351 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:10.939921 24351 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:10.942380 24351 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:10.943223 24351 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:10.946141 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 502 index: 328. Preceding OpId from leader: term: 503 index: 330. (index mismatch)
I1014 19:27:10.946758 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 503 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 502 index: 328. Preceding OpId from leader: term: 503 index: 330. (index mismatch)
I1014 19:27:10.946799 23414 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I1014 19:27:10.947206 23414 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I1014 19:27:10.948511 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.948789 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.950372 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.950587 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.952639 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.952844 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.953186 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.953351 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:10.958323 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.958547 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:10.958767 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 503 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:10.958925 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.133713 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 503 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:11.133819 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:11.133884 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 503 FOLLOWER]: Advancing to term 504
I1014 19:27:11.135931 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.136463 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 504 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.136626 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 504 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:11.137208 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 504 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:11.137410 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 504 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.137984 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 504 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.138078 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 504 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 504 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.138175 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 504 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.138514 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 504 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 504 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.138581 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 504 election: Election decided. Result: candidate lost.
I1014 19:27:11.138777 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:11.138839 24617 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Leader election lost for term 504. Reason: could not achieve majority
I1014 19:27:11.150024 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 503. Current term is 504. Ops: []
I1014 19:27:11.150569 24314 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 503.330, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I1014 19:27:11.150812 24314 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 LEADER]: Stepping down as leader of term 503
I1014 19:27:11.150848 24314 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:11.151051 24314 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 503.330, Last appended by leader: 330, Current term: 503, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.151226 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:11.151257 24314 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:11.151367 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 503 FOLLOWER]: Advancing to term 504
I1014 19:27:11.232079 24607 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:11.232149 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (starting election)
I1014 19:27:11.232204 24607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 503 FOLLOWER]: Advancing to term 504
I1014 19:27:11.233943 24607 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.234323 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 504 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.234480 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 504 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.235119 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 504 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.235303 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 504: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:11.235318 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 504 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.235504 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (vote granted)
I1014 19:27:11.235743 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 504 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 504: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:11.237498 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 504 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 504.
I1014 19:27:11.237844 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 504 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.237908 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 504 election: Election decided. Result: candidate won.
I1014 19:27:11.238579 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:11.238651 24607 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 FOLLOWER]: Leader election won for term 504
I1014 19:27:11.255131 24607 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:11.255494 24607 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 503.330, Last appended by leader: 330, Current term: 504, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.258008 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 504 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.258267 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.279127 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:11.279203 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.165s (starting election)
I1014 19:27:11.279258 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 504 FOLLOWER]: Advancing to term 505
I1014 19:27:11.281363 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.281868 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 505 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.282033 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 505 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:11.282907 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 505 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:11.283095 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 505 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.283569 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 505 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 505 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.284646 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 505 candidate_status { last_received { term: 503 index: 330 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.284802 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 504 FOLLOWER]: Advancing to term 505
I1014 19:27:11.284869 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:27:11.286717 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 505.
I1014 19:27:11.287094 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 505 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.287149 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 505 election: Election decided. Result: candidate won.
I1014 19:27:11.287501 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:11.287618 24617 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 FOLLOWER]: Leader election won for term 505
I1014 19:27:11.287739 24617 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:11.287956 24617 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 503.330, Last appended by leader: 330, Current term: 505, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.289065 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.289225 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.337774 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 504. Current term is 505. Ops: []
I1014 19:27:11.338711 24757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I1014 19:27:11.338953 24757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 LEADER]: Stepping down as leader of term 504
I1014 19:27:11.338994 24757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:11.339167 24757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 330, Committed index: 330, Last appended: 504.331, Last appended by leader: 331, Current term: 504, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.339355 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:11.339397 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:11.339534 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 504 FOLLOWER]: Advancing to term 505
I1014 19:27:11.376960 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 503 index: 330. Preceding OpId from leader: term: 505 index: 331. (index mismatch)
I1014 19:27:11.377405 22681 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I1014 19:27:11.378875 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.379071 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.379889 22681 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.380074 22681 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.380550 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 505 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 504 index: 331. Preceding OpId from leader: term: 505 index: 331. (term mismatch)
I1014 19:27:11.380602 29010 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 330
I1014 19:27:11.380625 29010 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 504.331
I1014 19:27:11.380664 29010 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 505 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:11.381218 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.381422 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.381408 24617 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 331, Last known committed idx: 330, Time since last communication: 0.000s
I1014 19:27:11.382941 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.383175 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.383301 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.383525 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.483270 24757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 505 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:11.483350 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:11.483403 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 505 FOLLOWER]: Advancing to term 506
I1014 19:27:11.485462 24757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.485960 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 506 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.486106 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 506 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.487169 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 506 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.487407 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 506 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.487519 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 506 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.487699 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 506 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.488071 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 506 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 506 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.489184 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 505 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.489367 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.489621 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 505. Current term is 506. Ops: []
I1014 19:27:11.490133 22681 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 505.331, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I1014 19:27:11.490309 22681 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 LEADER]: Stepping down as leader of term 505
I1014 19:27:11.490346 22681 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:11.490492 22681 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 505.331, Last appended by leader: 331, Current term: 505, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.490676 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:11.490725 22681 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:11.490747 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 506 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 506 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.490818 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 506 election: Election decided. Result: candidate lost.
I1014 19:27:11.490831 22681 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 505 FOLLOWER]: Advancing to term 506
I1014 19:27:11.491235 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:11.491291 24757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Leader election lost for term 506. Reason: could not achieve majority
I1014 19:27:11.557533 24314 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:11.557605 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:11.557648 24314 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 505 FOLLOWER]: Advancing to term 506
I1014 19:27:11.559619 24314 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.560066 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 506 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.560235 24314 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 506 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:11.561496 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 506 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:11.561538 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 506 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.561707 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 506: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:11.561687 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (vote granted)
I1014 19:27:11.562913 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 506 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 506: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:11.563567 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 506 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 506.
I1014 19:27:11.563890 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 506 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.563941 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 506 election: Election decided. Result: candidate won.
I1014 19:27:11.564615 24314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:11.564679 24314 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 FOLLOWER]: Leader election won for term 506
I1014 19:27:11.565296 24314 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:11.565556 24314 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 505.331, Last appended by leader: 331, Current term: 506, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.566954 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 506 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.567131 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.574498 24888 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:11.576383 24888 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:11.639335 24757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:11.639437 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.171s (starting election)
I1014 19:27:11.639492 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 506 FOLLOWER]: Advancing to term 507
I1014 19:27:11.641587 24757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.642071 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 507 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.642220 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 507 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.643033 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 507 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.643244 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 507 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.643815 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 507 candidate_status { last_received { term: 505 index: 331 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.643959 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 506 FOLLOWER]: Advancing to term 507
I1014 19:27:11.644024 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:11.644817 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 506. Current term is 507. Ops: []
I1014 19:27:11.645651 23414 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I1014 19:27:11.645874 23414 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 LEADER]: Stepping down as leader of term 506
I1014 19:27:11.645911 23414 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:11.646040 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 507 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 507.
I1014 19:27:11.646080 23414 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 506.332, Last appended by leader: 332, Current term: 506, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.646255 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:11.646293 23414 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:11.646389 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 FOLLOWER]: Advancing to term 507
I1014 19:27:11.647244 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 507 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 506 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 507 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.647574 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 507 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.647622 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 507 election: Election decided. Result: candidate won.
I1014 19:27:11.647972 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.153s (election complete)
I1014 19:27:11.648108 24757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 FOLLOWER]: Leader election won for term 507
I1014 19:27:11.648406 24757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:11.648779 24757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 505.331, Last appended by leader: 331, Current term: 507, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.650791 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.650959 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.736013 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 507 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 505 index: 331. Preceding OpId from leader: term: 507 index: 332. (index mismatch)
I1014 19:27:11.736758 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I1014 19:27:11.738351 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.738567 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.740067 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.740260 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.741447 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 507 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 506 index: 332. Preceding OpId from leader: term: 507 index: 332. (term mismatch)
I1014 19:27:11.741494 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 331
I1014 19:27:11.741513 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 506.332
I1014 19:27:11.741546 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 507 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:11.742449 24607 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I1014 19:27:11.743978 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.744200 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.744333 24314 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.744508 24314 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:11.746006 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.746212 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:11.765888 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.545s	user 0.001s	sys 0.001s
I1014 19:27:11.765938 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:11.766896 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:11.776167 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 507 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:11.776384 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:11.795995 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:11.797914 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:11.897009 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 507 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:11.897090 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:11.897143 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 507 FOLLOWER]: Advancing to term 508
I1014 19:27:11.899092 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.899633 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 508 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.899796 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 508 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:11.900642 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 508 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:11.900840 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 508 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.901242 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 507 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:11.901304 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:11.901250 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 508 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.901347 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 507 FOLLOWER]: Advancing to term 508
I1014 19:27:11.902431 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 508 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 508 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.903414 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 508 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.903750 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 508 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 508: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:11.903906 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 508 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.904042 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 508 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:11.904165 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 508 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 508 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 508: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:11.904228 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 508 election: Election decided. Result: candidate lost.
I1014 19:27:11.904747 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 508 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:11.904922 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 508 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.905210 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 508 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.905369 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 508: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:11.905405 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 508 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 508 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:11.905679 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:11.905730 24617 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Leader election lost for term 508. Reason: could not achieve majority
I1014 19:27:11.906551 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 508 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 508: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:11.906612 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 508 election: Election decided. Result: candidate lost.
I1014 19:27:11.906811 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:11.906865 23414 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 508 FOLLOWER]: Leader election lost for term 508. Reason: could not achieve majority
I1014 19:27:11.910749 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 507. Current term is 508. Ops: []
I1014 19:27:11.911386 24607 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 507.333, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I1014 19:27:11.911602 24757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Stepping down as leader of term 507
I1014 19:27:11.911644 24757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:11.911813 24757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 507.333, Last appended by leader: 333, Current term: 507, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.911991 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:11.912027 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:11.912148 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 507 FOLLOWER]: Advancing to term 508
I1014 19:27:11.993494 24757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 508 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:11.993572 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:11.993625 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 508 FOLLOWER]: Advancing to term 509
I1014 19:27:11.995782 24757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:11.996290 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 509 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:11.996955 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 509 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:11.997108 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 508 FOLLOWER]: Advancing to term 509
I1014 19:27:11.997176 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.213s (vote granted)
I1014 19:27:11.997689 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 509 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:11.998294 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 509 candidate_status { last_received { term: 507 index: 333 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:11.998461 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 508 FOLLOWER]: Advancing to term 509
I1014 19:27:11.998526 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.151s (vote granted)
I1014 19:27:11.999150 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 509 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 509.
I1014 19:27:12.000475 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 509.
I1014 19:27:12.000916 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 509 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.000972 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 509 election: Election decided. Result: candidate won.
I1014 19:27:12.001171 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 509 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.001400 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:12.001523 24757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 FOLLOWER]: Leader election won for term 509
I1014 19:27:12.001652 24757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:12.001875 24757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 507.333, Last appended by leader: 333, Current term: 509, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.002967 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.003150 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
W1014 19:27:12.060796 23129 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.487s	user 0.002s	sys 0.000s
I1014 19:27:12.060843 23129 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:12.061801 23129 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:12.062815 23129 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 502 vs 509: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:12.063043 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:12.063177 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:12.063315 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:12.063486 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:12.063611 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 509 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.078030 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:12.078860 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:12.081806 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 509 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 507 index: 333. Preceding OpId from leader: term: 509 index: 335. (index mismatch)
I1014 19:27:12.082223 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 507 index: 333. Preceding OpId from leader: term: 509 index: 335. (index mismatch)
I1014 19:27:12.083040 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I1014 19:27:12.084806 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.085001 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.085700 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I1014 19:27:12.087601 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.087802 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.087779 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.087962 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.088546 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:12.089448 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.089653 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.090270 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:12.091105 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.091297 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.091944 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.092133 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.137764 24888 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:12.138379 24888 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:12.139412 24888 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 506 vs 509: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 509 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.191684 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 509 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:12.191762 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:12.191821 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 509 FOLLOWER]: Advancing to term 510
I1014 19:27:12.193905 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 510 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.194402 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 510 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.194602 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 510 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.195693 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 510 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.195880 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 510 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.196395 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 510 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 509. Current term is 510. Ops: []
I1014 19:27:12.196360 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 510 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.196570 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 510 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.196831 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 510 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 510 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.196950 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 510 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 510 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.197031 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 510 election: Election decided. Result: candidate lost.
I1014 19:27:12.197211 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:12.197271 24617 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 510 FOLLOWER]: Leader election lost for term 510. Reason: could not achieve majority
I1014 19:27:12.197476 24896 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 509.335, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I1014 19:27:12.197715 24896 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 LEADER]: Stepping down as leader of term 509
I1014 19:27:12.197754 24896 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:12.197928 24896 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 509.335, Last appended by leader: 335, Current term: 509, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.198117 24896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:12.198171 24896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:12.198307 24896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 509 FOLLOWER]: Advancing to term 510
I1014 19:27:12.281656 24894 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 510 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:12.281731 24894 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:12.281785 24894 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 510 FOLLOWER]: Advancing to term 511
I1014 19:27:12.283879 24894 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.284376 24894 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 511 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.284548 24894 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 511 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.284955 23414 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:12.285020 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:12.285068 23414 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 509 FOLLOWER]: Advancing to term 510
I1014 19:27:12.285465 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 511 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.285624 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 510 FOLLOWER]: Advancing to term 511
I1014 19:27:12.285681 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (vote granted)
I1014 19:27:12.287133 23414 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 510 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.287612 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 510 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.287637 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 511.
I1014 19:27:12.287755 23414 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 510 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.288048 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 511 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.288174 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 510 FOLLOWER]: Advancing to term 511
I1014 19:27:12.288230 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:12.289503 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 511 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.289563 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 511 election: Election decided. Result: candidate won.
I1014 19:27:12.290092 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 511 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 511.
I1014 19:27:12.290050 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 510 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.290213 28639 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 510. Current term is 511.
I1014 19:27:12.290572 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 510 candidate_status { last_received { term: 509 index: 335 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.290741 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 510. Current term is 511.
W1014 19:27:12.290766 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 510 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 510. Current term is 511.
I1014 19:27:12.290793 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 510 election: Cancelling election due to peer responding with higher term
I1014 19:27:12.291056 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 511 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:27:12.291609 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 510 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 510. Current term is 511.
I1014 19:27:12.291811 23414 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (election complete)
I1014 19:27:12.291867 23414 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 511 FOLLOWER]: Leader election lost for term 510. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 510. Current term is 511.
I1014 19:27:12.292004 24894 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:12.292066 24894 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 FOLLOWER]: Leader election won for term 511
I1014 19:27:12.292737 24894 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:12.292989 24894 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 509.335, Last appended by leader: 335, Current term: 511, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.294183 24896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.294394 24896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.380442 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 509 index: 335. Preceding OpId from leader: term: 511 index: 336. (index mismatch)
I1014 19:27:12.380987 24894 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I1014 19:27:12.382361 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.382585 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.384243 24894 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.384451 24894 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.385387 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 511 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 509 index: 335. Preceding OpId from leader: term: 511 index: 336. (index mismatch)
I1014 19:27:12.385942 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.386132 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.386921 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I1014 19:27:12.390502 23414 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.390708 23414 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.390745 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.390992 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.486243 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:12.486317 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:12.486364 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 511 FOLLOWER]: Advancing to term 512
I1014 19:27:12.488067 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.488606 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 512 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.488700 25032 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 511 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:12.488766 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:12.488775 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 512 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.488816 25032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 511 FOLLOWER]: Advancing to term 512
I1014 19:27:12.491441 25032 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.491916 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 512 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.492053 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 512 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.492909 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 512 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.493129 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 512 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.493741 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 511 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.493965 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.494305 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 511. Current term is 512. Ops: []
I1014 19:27:12.494645 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 512 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.494828 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 512: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:12.495000 24896 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 511.336, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I1014 19:27:12.495208 24896 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 LEADER]: Stepping down as leader of term 511
I1014 19:27:12.495242 24896 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:12.495445 24896 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 511.336, Last appended by leader: 336, Current term: 511, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.495633 24896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:12.495676 24896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:12.495810 24896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 FOLLOWER]: Advancing to term 512
I1014 19:27:12.495986 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 512 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 511 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 512 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.496155 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 512 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.496299 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 512: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:12.496666 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 512 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 512: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:12.498152 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 512 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 512: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:12.498214 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 512 election: Election decided. Result: candidate lost.
I1014 19:27:12.499176 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 512 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.499320 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (vote granted)
I1014 19:27:12.500633 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 512 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 512.
I1014 19:27:12.500984 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 512 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.501045 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 512 election: Election decided. Result: candidate won.
I1014 19:27:12.501272 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:12.501365 25032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election lost for term 512. Reason: could not achieve majority
I1014 19:27:12.501902 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:12.501966 24617 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 FOLLOWER]: Leader election won for term 512
I1014 19:27:12.502809 24617 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:12.503010 24617 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 511.336, Last appended by leader: 336, Current term: 512, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.504135 25033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 512 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.504627 25033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.504858 23129 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:12.506742 23129 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:12.594144 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 511 index: 336. Preceding OpId from leader: term: 512 index: 337. (index mismatch)
I1014 19:27:12.595394 25033 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I1014 19:27:12.598924 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 512 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 511 index: 336. Preceding OpId from leader: term: 512 index: 337. (index mismatch)
I1014 19:27:12.598939 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 512 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.599117 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.599503 25033 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I1014 19:27:12.600363 25033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 512 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.600623 25033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.600910 24757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 512 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:12.600971 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:12.601024 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 512 FOLLOWER]: Advancing to term 513
I1014 19:27:12.601033 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 512 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.601210 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.602684 24757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 513 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.602952 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 513 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 512. Current term is 513. Ops: []
I1014 19:27:12.603169 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 513 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.603327 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 513 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.603840 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 513 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.603893 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 513 candidate_status { last_received { term: 511 index: 336 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.604009 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.604048 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.604228 24617 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 511.336, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s
I1014 19:27:12.604310 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 513 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.604452 24617 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 LEADER]: Stepping down as leader of term 512
I1014 19:27:12.604488 24617 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:12.604660 24617 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 512.337, Last appended by leader: 337, Current term: 512, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.604823 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:12.604862 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:12.604984 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 FOLLOWER]: Advancing to term 513
I1014 19:27:12.607728 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 513 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 512 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.607797 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 513 election: Election decided. Result: candidate lost.
I1014 19:27:12.608104 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:12.608175 24757 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 513 FOLLOWER]: Leader election lost for term 513. Reason: could not achieve majority
I1014 19:27:12.684087 25033 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 513 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:12.684165 25033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (starting election)
I1014 19:27:12.684224 25033 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 513 FOLLOWER]: Advancing to term 514
I1014 19:27:12.686506 25033 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.686949 25033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 514 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.687069 25033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 514 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.687562 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 514 candidate_status { last_received { term: 512 index: 337 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.687731 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 514 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.687916 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 514 candidate_status { last_received { term: 512 index: 337 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.688063 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 513 FOLLOWER]: Advancing to term 514
I1014 19:27:12.688088 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 514 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 514 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.688140 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:27:12.689702 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 514.
I1014 19:27:12.690248 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 514 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.690292 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 514 election: Election decided. Result: candidate won.
I1014 19:27:12.691073 25033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (election complete)
I1014 19:27:12.691126 25033 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 FOLLOWER]: Leader election won for term 514
I1014 19:27:12.691295 25033 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:12.691505 25033 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 337, Committed index: 337, Last appended: 512.337, Last appended by leader: 337, Current term: 514, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.693207 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.693435 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.701709 25032 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:12.701788 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:12.701834 25032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 512 FOLLOWER]: Advancing to term 513
I1014 19:27:12.703970 25032 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 513 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.704438 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 513 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.704612 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 513 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.705706 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 513 candidate_status { last_received { term: 512 index: 337 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.705863 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.706143 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 513 candidate_status { last_received { term: 512 index: 337 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
W1014 19:27:12.706271 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 513 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.706301 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 513 election: Cancelling election due to peer responding with higher term
I1014 19:27:12.706329 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 513. Current term is 514.
I1014 19:27:12.706511 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:12.706594 25032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 513 FOLLOWER]: Advancing to term 514
W1014 19:27:12.707079 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 513 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 513. Current term is 514.
I1014 19:27:12.708678 25032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 514 FOLLOWER]: Leader election lost for term 513. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 513 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.776223 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 514 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 512 index: 337. Preceding OpId from leader: term: 514 index: 338. (index mismatch)
I1014 19:27:12.776752 25033 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 337, Time since last communication: 0.000s
I1014 19:27:12.778012 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.778204 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.779352 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.779573 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.779937 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.780125 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.780638 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 511 index: 336. Preceding OpId from leader: term: 514 index: 338. (index mismatch)
I1014 19:27:12.781811 24617 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 338, Last known committed idx: 336, Time since last communication: 0.000s
I1014 19:27:12.783711 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.783766 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.783906 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.784060 24896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.784226 24896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.783915 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.880867 25032 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 514 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:12.880944 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:12.880992 25032 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 514 FOLLOWER]: Advancing to term 515
I1014 19:27:12.883165 25032 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.883714 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 515 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.883877 25032 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 515 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:12.884125 24757 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:12.884204 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:12.884258 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 514 FOLLOWER]: Advancing to term 515
I1014 19:27:12.884439 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 514 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.884636 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.885067 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 515 candidate_status { last_received { term: 514 index: 338 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:12.885712 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 515 candidate_status { last_received { term: 514 index: 338 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.885895 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 515 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.886575 24757 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.886919 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 515 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 515 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:12.887133 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 515: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:12.887181 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 515 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.887372 24757 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 515 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:12.887542 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 514. Current term is 515. Ops: []
I1014 19:27:12.888010 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 515 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 515: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:12.888073 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 515 election: Election decided. Result: candidate lost.
I1014 19:27:12.888412 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 515 candidate_status { last_received { term: 514 index: 338 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:12.888594 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 515: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:12.889266 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 515 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 515: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:12.889497 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:12.889400 25033 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 514.338, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I1014 19:27:12.889559 25032 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Leader election lost for term 515. Reason: could not achieve majority
I1014 19:27:12.889647 25033 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Stepping down as leader of term 514
I1014 19:27:12.889686 25033 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:12.889883 25033 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 514.338, Last appended by leader: 338, Current term: 514, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.890054 25033 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:12.890095 25033 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:12.890225 25033 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 514 FOLLOWER]: Advancing to term 515
I1014 19:27:12.892627 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 515 candidate_status { last_received { term: 514 index: 338 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:12.892771 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (vote granted)
I1014 19:27:12.894752 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 515 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 515.
I1014 19:27:12.895424 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 515 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:12.895498 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 515 election: Election decided. Result: candidate won.
I1014 19:27:12.895793 24757 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:12.895860 24757 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Leader election won for term 515
I1014 19:27:12.896044 24757 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:12.896279 24757 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 338, Committed index: 338, Last appended: 514.338, Last appended by leader: 338, Current term: 515, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:12.903874 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.904115 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.975477 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 515 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 514 index: 338. Preceding OpId from leader: term: 515 index: 339. (index mismatch)
I1014 19:27:12.977893 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I1014 19:27:12.979416 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.979605 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.981865 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.982059 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:12.983129 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 514 index: 338. Preceding OpId from leader: term: 515 index: 339. (index mismatch)
I1014 19:27:12.983498 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.983665 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.983939 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 339, Last known committed idx: 338, Time since last communication: 0.000s
I1014 19:27:12.986469 25032 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.986682 25032 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:12.986724 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:12.987490 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.064673 24896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 515 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.064914 24896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.263387 25163 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:13.263492 25163 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:13.263550 25163 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 515 FOLLOWER]: Advancing to term 516
I1014 19:27:13.265758 25163 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 516 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.266222 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 516 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:13.266368 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 516 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.266857 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 515 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:13.266930 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:13.266979 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 515 FOLLOWER]: Advancing to term 516
I1014 19:27:13.266976 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 516 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:13.267779 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 516 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.267961 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 516 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.268312 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 516 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 516 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.268723 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.269129 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 516 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.269222 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 516: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:13.269271 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 516 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.269722 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 516 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.269892 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 516 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 516: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:13.269925 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 516 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 516: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:13.269985 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 516 election: Election decided. Result: candidate lost.
I1014 19:27:13.270048 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 516 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.270195 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 516 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.270364 25163 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:13.270438 25163 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 516 FOLLOWER]: Leader election lost for term 516. Reason: could not achieve majority
I1014 19:27:13.270509 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 516 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 516 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 516: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:13.270707 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 516 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 516 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.270762 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 516 election: Election decided. Result: candidate lost.
I1014 19:27:13.270946 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:27:13.270997 24617 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Leader election lost for term 516. Reason: could not achieve majority
I1014 19:27:13.274674 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 515. Current term is 516. Ops: []
I1014 19:27:13.275169 24757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 515.339, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I1014 19:27:13.275388 24757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Stepping down as leader of term 515
I1014 19:27:13.275449 24757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:13.275610 24757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 515.339, Last appended by leader: 339, Current term: 515, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.275784 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:13.275820 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:13.275933 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 515 FOLLOWER]: Advancing to term 516
W1014 19:27:13.352340 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.262s	user 0.003s	sys 0.000s
I1014 19:27:13.352389 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:13.353106 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:13.354096 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 509 vs 516: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 516 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.355590 24607 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 516 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:13.355651 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.146s (starting election)
I1014 19:27:13.355696 24607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 516 FOLLOWER]: Advancing to term 517
W1014 19:27:13.356920 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 516 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.357614 24607 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.358095 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 517 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
W1014 19:27:13.358244 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 517 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.358261 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 517 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:27:13.358660 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 517 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:13.358950 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 517 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.359051 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 517 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:13.359189 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 516 FOLLOWER]: Advancing to term 517
I1014 19:27:13.359244 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.175s (vote granted)
W1014 19:27:13.359299 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 517 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.359652 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 517 candidate_status { last_received { term: 515 index: 339 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.359802 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 516 FOLLOWER]: Advancing to term 517
I1014 19:27:13.359865 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.160s (vote granted)
I1014 19:27:13.361240 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 517 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 517.
I1014 19:27:13.361665 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 517 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:13.361721 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 517 election: Election decided. Result: candidate won.
I1014 19:27:13.361893 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:13.361925 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 517.
I1014 19:27:13.361946 24607 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 FOLLOWER]: Leader election won for term 517
I1014 19:27:13.362015 24607 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
W1014 19:27:13.362176 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 517 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.362213 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 517 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.362270 24607 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 515.339, Last appended by leader: 339, Current term: 517, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.363188 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.363504 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.363891 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:13.365900 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:13.446173 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 515 index: 339. Preceding OpId from leader: term: 517 index: 340. (index mismatch)
I1014 19:27:13.446768 24757 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I1014 19:27:13.449093 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.449282 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.450451 24757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.450659 24757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.451552 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 517 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 515 index: 339. Preceding OpId from leader: term: 517 index: 340. (index mismatch)
I1014 19:27:13.451980 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.452145 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.452832 24607 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I1014 19:27:13.458894 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.458865 25293 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.459079 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.459106 25293 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.554015 25293 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 517 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:13.554109 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:13.554177 25293 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 517 FOLLOWER]: Advancing to term 518
I1014 19:27:13.556349 25293 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.556864 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 518 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.557024 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 518 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.558034 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 518 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.558221 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 518 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.558193 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 518 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.558377 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 518 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.558676 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 518 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 518 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.558771 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 518 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 518 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.558831 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 518 election: Election decided. Result: candidate lost.
I1014 19:27:13.559047 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:13.559113 25293 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Leader election lost for term 518. Reason: could not achieve majority
I1014 19:27:13.571530 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 517 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.571760 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.572453 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 517. Current term is 518. Ops: []
I1014 19:27:13.573633 24757 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 517.340, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.573873 24757 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 LEADER]: Stepping down as leader of term 517
I1014 19:27:13.573913 24757 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:13.574081 24757 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 517.340, Last appended by leader: 340, Current term: 517, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.574281 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:13.574323 24757 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:13.574479 24757 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 517 FOLLOWER]: Advancing to term 518
I1014 19:27:13.644848 25163 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:13.644938 25163 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:13.645004 25163 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 517 FOLLOWER]: Advancing to term 518
I1014 19:27:13.647245 25163 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.647773 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 518 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:13.647969 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 518 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.648490 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 518 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:13.648679 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 518: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:13.648810 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 518 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.648980 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:13.650643 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 518 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 518.
I1014 19:27:13.650908 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 518 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 518: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:13.650998 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 518 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.651069 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 518 election: Election decided. Result: candidate won.
I1014 19:27:13.651320 25163 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:13.651444 25163 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 FOLLOWER]: Leader election won for term 518
I1014 19:27:13.651825 25163 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:13.652067 25163 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 517.340, Last appended by leader: 340, Current term: 518, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.653970 25297 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 518 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.654214 25297 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.655519 25298 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:13.657361 25298 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:13.662091 25293 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:13.662153 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.223s (starting election)
I1014 19:27:13.662199 25293 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 518 FOLLOWER]: Advancing to term 519
I1014 19:27:13.663638 25293 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.664044 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 519 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.664188 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 519 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.665262 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 519 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.665458 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 519 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.665490 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 519 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.665647 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 518 FOLLOWER]: Advancing to term 519
I1014 19:27:13.665715 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:13.665912 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 519 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 519 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.667866 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 519 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 519.
I1014 19:27:13.668299 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 519 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.668354 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 519 election: Election decided. Result: candidate won.
I1014 19:27:13.668568 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:13.668628 25293 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 FOLLOWER]: Leader election won for term 519
I1014 19:27:13.668716 25293 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:13.668905 25293 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 517.340, Last appended by leader: 340, Current term: 519, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.670604 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 519 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.670792 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.752933 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 518. Current term is 519. Ops: []
I1014 19:27:13.753494 25297 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.753706 25297 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 519.341 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 340
I1014 19:27:13.753798 25163 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 LEADER]: Stepping down as leader of term 518
I1014 19:27:13.753855 25163 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:13.754060 25163 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 518.341, Last appended by leader: 341, Current term: 518, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.754261 25163 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:13.754312 25163 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:13.754415 25163 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 518 FOLLOWER]: Advancing to term 519
I1014 19:27:13.773578 24607 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 519 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:13.773659 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:27:13.773720 24607 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 519 FOLLOWER]: Advancing to term 520
I1014 19:27:13.775815 24607 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 520 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.776283 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 520 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:13.776453 24607 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 520 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.777053 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 520 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:13.777238 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 520 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.777323 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 520 candidate_status { last_received { term: 517 index: 340 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.777532 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 519 FOLLOWER]: Advancing to term 520
I1014 19:27:13.777575 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 520 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 520 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.779075 28826 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 520 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 520 because replica has last-logged OpId of term: 518 index: 341, which is greater than that of the candidate, which has last-logged OpId of term: 517 index: 340.
I1014 19:27:13.779592 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 520 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 519. Current term is 520. Ops: []
I1014 19:27:13.779675 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 520 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 520 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 520 because replica has last-logged OpId of term: 518 index: 341, which is greater than that of the candidate, which has last-logged OpId of term: 517 index: 340.
I1014 19:27:13.779739 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 520 election: Election decided. Result: candidate lost.
I1014 19:27:13.780078 24607 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:13.780131 24607 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 520 FOLLOWER]: Leader election lost for term 520. Reason: could not achieve majority
I1014 19:27:13.780048 24617 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.780225 24617 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 518.341 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 340
I1014 19:27:13.780344 24617 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Stepping down as leader of term 519
I1014 19:27:13.780390 24617 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:13.780593 24617 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 519.341, Last appended by leader: 341, Current term: 519, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.780786 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:13.780838 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:13.780951 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 519 FOLLOWER]: Advancing to term 520
I1014 19:27:13.857659 25293 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 520 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:13.857765 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:13.857820 25293 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 520 FOLLOWER]: Advancing to term 521
I1014 19:27:13.859876 25293 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.860227 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 521 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.860352 25293 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 521 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.860847 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 521 candidate_status { last_received { term: 519 index: 341 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:13.861012 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 520 FOLLOWER]: Advancing to term 521
I1014 19:27:13.861013 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 521 candidate_status { last_received { term: 519 index: 341 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.861095 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (vote granted)
I1014 19:27:13.861181 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 520 FOLLOWER]: Advancing to term 521
I1014 19:27:13.861258 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.156s (vote granted)
I1014 19:27:13.863143 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 521 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 521.
I1014 19:27:13.863380 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 521 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 521.
I1014 19:27:13.863728 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 521 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.863785 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 521 election: Election decided. Result: candidate won.
I1014 19:27:13.863967 25293 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:13.864141 25293 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 FOLLOWER]: Leader election won for term 521
I1014 19:27:13.864364 25293 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:13.864740 25293 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 519.341, Last appended by leader: 341, Current term: 521, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.866339 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 521 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.866549 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.868788 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 521 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:13.969285 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 521 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 518 index: 341. Preceding OpId from leader: term: 521 index: 342. (index mismatch)
I1014 19:27:13.971658 25293 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.971817 25293 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 518.341 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 340
I1014 19:27:13.974097 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 340
I1014 19:27:13.974144 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 518.341
I1014 19:27:13.974205 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 521 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:13.974866 25163 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 521 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:13.974975 25163 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:13.975077 25163 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 521 FOLLOWER]: Advancing to term 522
I1014 19:27:13.977280 25163 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.977780 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 522 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:13.978140 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 521. Current term is 522. Ops: []
I1014 19:27:13.978142 25297 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 522 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.978145 25163 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 522 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.978396 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 522 candidate_status { last_received { term: 521 index: 342 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:13.978360 25297 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:13.978880 24617 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 521.342, Next index: 343, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.979055 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 522 candidate_status { last_received { term: 521 index: 342 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:13.979223 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 521 FOLLOWER]: Advancing to term 522
I1014 19:27:13.979295 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.172s (vote granted)
I1014 19:27:13.981609 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 522 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 522.
I1014 19:27:13.981930 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 522 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:13.981997 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 522 election: Election decided. Result: candidate won.
I1014 19:27:13.982206 25297 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:13.982280 25297 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 FOLLOWER]: Leader election won for term 522
I1014 19:27:13.982383 25297 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:13.982569 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 522 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 521. Current term is 522. Ops: []
I1014 19:27:13.982625 25297 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 521.342, Last appended by leader: 342, Current term: 522, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.983530 25297 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 522 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.983722 25297 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.992498 25426 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 342, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:13.993088 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 522 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.993264 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 521 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:13.993470 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:13.993674 25293 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 LEADER]: Stepping down as leader of term 521
I1014 19:27:13.993737 25293 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:13.993718 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 522 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 522 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:13.993978 25293 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 521.342, Last appended by leader: 342, Current term: 521, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:13.994241 25293 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:13.994354 25293 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:13.994544 25293 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 521 FOLLOWER]: Advancing to term 522
I1014 19:27:14.074074 25426 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 522 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:14.074144 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:14.074194 25426 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 522 FOLLOWER]: Advancing to term 523
I1014 19:27:14.076138 25426 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.076571 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 523 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:14.076726 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 523 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.077328 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 523 candidate_status { last_received { term: 521 index: 342 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:14.077347 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 523 candidate_status { last_received { term: 521 index: 342 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:14.077491 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 522 FOLLOWER]: Advancing to term 523
I1014 19:27:14.077549 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 523 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.077586 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:14.078073 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 523 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 523 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.079159 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 523 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 523.
I1014 19:27:14.079429 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 523 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.079483 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 523 election: Election decided. Result: candidate won.
I1014 19:27:14.079658 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:14.079710 25426 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 FOLLOWER]: Leader election won for term 523
I1014 19:27:14.079790 25426 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:14.079995 25426 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 521.342, Last appended by leader: 342, Current term: 523, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.080809 25293 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.080987 25293 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.089363 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 523 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 522. Current term is 523. Ops: []
I1014 19:27:14.089884 25163 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 343, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:14.090127 25297 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 LEADER]: Stepping down as leader of term 522
I1014 19:27:14.090181 25297 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:14.090353 25297 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 522.343, Last appended by leader: 343, Current term: 522, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.090498 25297 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:14.090590 25297 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:14.090704 25297 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 522 FOLLOWER]: Advancing to term 523
I1014 19:27:14.144313 25298 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:14.145267 25298 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:14.146664 25298 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 518 vs 523: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 523 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.146988 25298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 523 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.158824 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 523 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 517 index: 340. Preceding OpId from leader: term: 523 index: 343. (index mismatch)
I1014 19:27:14.159301 25293 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:14.162451 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.162446 25431 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.162688 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.162721 25431 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.162811 24607 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.163007 24607 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.163760 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.163976 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.164189 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 523 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 522 index: 343. Preceding OpId from leader: term: 523 index: 343. (term mismatch)
I1014 19:27:14.164258 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 342
I1014 19:27:14.164306 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 522.343
I1014 19:27:14.164371 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 523 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:14.165127 25426 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 340, Time since last communication: 0.000s
I1014 19:27:14.165794 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.166208 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.168426 25297 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.168658 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.168777 25297 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.168772 25434 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.169143 25434 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.169265 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.168452 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 523 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.171036 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.266504 25432 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 523 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:14.266597 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:14.266651 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 523 FOLLOWER]: Advancing to term 524
I1014 19:27:14.267918 25435 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 523 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:14.268002 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:14.268071 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 523 FOLLOWER]: Advancing to term 524
I1014 19:27:14.268908 25432 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.269182 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 523. Current term is 524. Ops: []
I1014 19:27:14.269433 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 524 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.269722 25426 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 523.343, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I1014 19:27:14.269934 25426 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 LEADER]: Stepping down as leader of term 523
I1014 19:27:14.269966 25426 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:14.270143 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 524 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:14.270130 25426 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 523.343, Last appended by leader: 343, Current term: 523, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.270318 25426 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:14.270352 25426 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:14.270474 25426 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 523 FOLLOWER]: Advancing to term 524
I1014 19:27:14.270730 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 524 candidate_status { last_received { term: 523 index: 343 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:14.271574 25435 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.271870 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 524: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:14.272336 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 524 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 524: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:14.272590 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 524 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.272785 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 524 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.273257 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 524 candidate_status { last_received { term: 523 index: 343 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.273290 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 524 candidate_status { last_received { term: 523 index: 343 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:14.273396 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:14.273355 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 524 candidate_status { last_received { term: 523 index: 343 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.273483 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 524: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:14.273871 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 524 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 524: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:14.275349 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 524.
I1014 19:27:14.275471 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 524 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:14.275985 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 524 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 524 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:14.276051 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 524 election: Election decided. Result: candidate lost.
I1014 19:27:14.276245 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:14.276309 25432 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Leader election lost for term 524. Reason: could not achieve majority
I1014 19:27:14.276545 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 524 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.276616 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 524 election: Election decided. Result: candidate won.
I1014 19:27:14.276916 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (election complete)
I1014 19:27:14.277024 25435 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 FOLLOWER]: Leader election won for term 524
I1014 19:27:14.277206 25435 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:14.277508 25435 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 523.343, Last appended by leader: 343, Current term: 524, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.278643 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.278839 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.279080 25298 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:14.280901 25298 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
W1014 19:27:14.336462 23129 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.830s	user 0.002s	sys 0.000s
I1014 19:27:14.336504 23129 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:14.337262 23129 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:14.338223 23129 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 512 vs 524: Illegal state: Replica 88e69d68a7914bf48d802a2f2e500b71 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.338475 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.338672 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.338876 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.339064 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.339238 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.339442 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.339630 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.339834 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.340030 23129 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 88e69d68a7914bf48d802a2f2e500b71 not leader. Raft Consensus state: current_term: 524 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.370777 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 523 index: 343. Preceding OpId from leader: term: 524 index: 344. (index mismatch)
I1014 19:27:14.371496 25163 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I1014 19:27:14.372968 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.373260 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.374049 25163 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.374310 25163 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.375569 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 523 index: 343. Preceding OpId from leader: term: 524 index: 344. (index mismatch)
I1014 19:27:14.376096 25297 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I1014 19:27:14.377377 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.377573 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.377667 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.377807 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.378031 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.378223 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:14.429889 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.064s	user 0.002s	sys 0.000s
I1014 19:27:14.429944 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:14.430932 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:14.432224 21475 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 517 vs 524: Illegal state: Replica 4043e04130264237805f012d8e251bd4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.432670 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.433465 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.433765 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.434154 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.434460 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.434767 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.435040 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.435312 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:14.435626 21475 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node 4043e04130264237805f012d8e251bd4 not leader. Raft Consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.477807 25426 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:14.477886 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:14.477937 25426 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 524 FOLLOWER]: Advancing to term 525
I1014 19:27:14.480121 25426 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.480648 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 525 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:14.480804 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 525 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.481412 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 525 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:14.481634 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 525 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.481994 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 525 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 525 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.481945 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 525 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:14.482141 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 525 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.482911 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 525 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 525 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.482975 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 525 election: Election decided. Result: candidate lost.
I1014 19:27:14.483382 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:14.483500 25426 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Leader election lost for term 525. Reason: could not achieve majority
I1014 19:27:14.497211 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 524 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.497499 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.497686 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 524. Current term is 525. Ops: []
I1014 19:27:14.498281 25435 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 524.344, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.498538 25434 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 LEADER]: Stepping down as leader of term 524
I1014 19:27:14.498600 25434 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:14.498803 25434 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 524.344, Last appended by leader: 344, Current term: 524, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.499008 25434 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:14.499066 25434 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:14.499199 25434 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 524 FOLLOWER]: Advancing to term 525
I1014 19:27:14.575302 25434 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 525 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:14.575325 25432 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:14.575395 25434 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:14.575417 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:14.575464 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 524 FOLLOWER]: Advancing to term 525
I1014 19:27:14.575465 25434 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 525 FOLLOWER]: Advancing to term 526
I1014 19:27:14.577811 25432 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 525 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.578349 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 525 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.578529 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 525 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:14.578830 25434 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.579351 25434 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 526 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.579480 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 525 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:14.579558 25434 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 526 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.579696 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 525. Current term is 526.
I1014 19:27:14.580039 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 525 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.580229 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 525: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:14.580513 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 526 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:14.580649 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 525 FOLLOWER]: Advancing to term 526
I1014 19:27:14.580718 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (vote granted)
W1014 19:27:14.582901 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 525 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 525. Current term is 526.
I1014 19:27:14.582908 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 526 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 526.
I1014 19:27:14.582943 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 525 election: Cancelling election due to peer responding with higher term
I1014 19:27:14.583312 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 526 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.583384 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 526 election: Election decided. Result: candidate won.
I1014 19:27:14.583406 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 526 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.583567 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Advancing to term 526
I1014 19:27:14.583637 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.218s (vote granted)
I1014 19:27:14.583686 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:14.583740 25432 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 526 FOLLOWER]: Leader election lost for term 525. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 525. Current term is 526.
I1014 19:27:14.583863 25434 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:14.583937 25434 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Leader election won for term 526
I1014 19:27:14.584038 25434 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:14.584250 25434 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 524.344, Last appended by leader: 344, Current term: 526, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.585260 25434 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 526 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.585538 25434 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.585731 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 526 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 526.
I1014 19:27:14.586166 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 526 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.586172 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 525 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 525 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 525: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:14.697144 25432 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 526 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:14.697226 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.196s (starting election)
I1014 19:27:14.697278 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 526 FOLLOWER]: Advancing to term 527
I1014 19:27:14.699434 25432 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.699965 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 527 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.700140 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 527 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:14.701848 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 527 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:14.702023 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 527 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.702095 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 526 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 524 index: 344. Preceding OpId from leader: term: 526 index: 345. (index mismatch)
I1014 19:27:14.702424 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 527 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 527 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.702396 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 527 candidate_status { last_received { term: 524 index: 344 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.702545 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 526 FOLLOWER]: Advancing to term 527
I1014 19:27:14.702616 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:14.704684 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 527.
I1014 19:27:14.705631 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 526. Current term is 527. Ops: []
I1014 19:27:14.705623 25435 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.706925 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 527 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.706984 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 527 election: Election decided. Result: candidate won.
I1014 19:27:14.707231 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 526. Current term is 527. Ops: []
I1014 19:27:14.707350 25435 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.707630 25435 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 LEADER]: Stepping down as leader of term 526
I1014 19:27:14.707721 25435 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:14.707854 25297 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 524.344, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.708012 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:14.708071 25432 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 FOLLOWER]: Leader election won for term 527
I1014 19:27:14.708573 25435 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 526.345, Last appended by leader: 345, Current term: 526, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.708760 25435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:14.708808 25435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:14.708925 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 526 FOLLOWER]: Advancing to term 527
I1014 19:27:14.714385 25432 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:14.717797 25432 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 524.344, Last appended by leader: 344, Current term: 527, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.719305 25566 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.719496 25566 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.719699 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:14.721649 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:14.798645 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 527 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 526 index: 345. Preceding OpId from leader: term: 527 index: 345. (term mismatch)
I1014 19:27:14.798712 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 344
I1014 19:27:14.798749 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 526.345
I1014 19:27:14.798808 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 527 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:14.799978 25566 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.801215 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.801522 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.803100 25566 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.803333 25566 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.804373 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 524 index: 344. Preceding OpId from leader: term: 527 index: 345. (index mismatch)
I1014 19:27:14.804519 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.804754 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.805294 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I1014 19:27:14.806845 25426 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.807090 25426 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.807267 25426 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.807431 25426 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:14.903187 25566 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 527 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:14.903443 25566 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:14.979817 25435 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 527 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:14.979907 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:14.979967 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 527 FOLLOWER]: Advancing to term 528
I1014 19:27:14.982214 25435 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.982729 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 528 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:14.982941 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 528 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:14.983455 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 528 candidate_status { last_received { term: 527 index: 345 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:14.983675 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 528 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.983851 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 528 candidate_status { last_received { term: 527 index: 345 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:14.984050 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 528 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.984154 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 528 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 528 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.984483 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 528 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 528 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:14.984582 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 528 election: Election decided. Result: candidate lost.
I1014 19:27:14.984961 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:14.985035 25435 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Leader election lost for term 528. Reason: could not achieve majority
I1014 19:27:14.987757 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 527. Current term is 528. Ops: []
I1014 19:27:14.988252 25432 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 527.345, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I1014 19:27:14.988487 25432 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 LEADER]: Stepping down as leader of term 527
I1014 19:27:14.988548 25432 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:14.988728 25432 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 527.345, Last appended by leader: 345, Current term: 527, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:14.988907 25432 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:14.988946 25432 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:14.989070 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 527 FOLLOWER]: Advancing to term 528
I1014 19:27:15.005332 25426 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:15.005429 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:15.005528 25426 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 527 FOLLOWER]: Advancing to term 528
I1014 19:27:15.007016 25426 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.007429 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 528 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.007620 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 528 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.008114 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 528 candidate_status { last_received { term: 527 index: 345 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.008327 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 528: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:15.008769 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 528 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 528: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:15.008821 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 528 candidate_status { last_received { term: 527 index: 345 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:15.008996 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:15.010478 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 528 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 528.
I1014 19:27:15.010812 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 528 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.010872 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 528 election: Election decided. Result: candidate won.
I1014 19:27:15.011091 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:15.011175 25426 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 FOLLOWER]: Leader election won for term 528
I1014 19:27:15.011297 25426 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:15.011481 25426 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 527.345, Last appended by leader: 345, Current term: 528, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.012338 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.012569 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.013345 25695 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:15.017529 25695 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:15.099092 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 527 index: 345. Preceding OpId from leader: term: 528 index: 346. (index mismatch)
I1014 19:27:15.099733 24617 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I1014 19:27:15.100982 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.101181 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.104861 25426 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.105082 25426 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.105585 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 528 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 527 index: 345. Preceding OpId from leader: term: 528 index: 346. (index mismatch)
I1014 19:27:15.106595 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.106827 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.107300 25426 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I1014 19:27:15.108738 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.108944 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.109084 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.109402 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.209147 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 528 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:15.209234 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:15.209287 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 528 FOLLOWER]: Advancing to term 529
I1014 19:27:15.211565 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.212101 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 529 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.213461 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 529 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.213582 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 529 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.213798 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 529 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.214349 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 529 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.214573 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 529 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.215016 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 529 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 529 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.215521 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 529 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 529 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.215579 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 529 election: Election decided. Result: candidate lost.
I1014 19:27:15.215937 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:15.216028 25651 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Leader election lost for term 529. Reason: could not achieve majority
I1014 19:27:15.217159 25426 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 528 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.217394 25426 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.218088 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 528. Current term is 529. Ops: []
I1014 19:27:15.218654 24617 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 528.346, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.218873 24617 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 LEADER]: Stepping down as leader of term 528
I1014 19:27:15.218919 24617 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:15.219094 24617 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 528.346, Last appended by leader: 346, Current term: 528, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.219277 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:15.219334 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:15.219480 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 528 FOLLOWER]: Advancing to term 529
I1014 19:27:15.297492 25435 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:15.297617 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:15.297685 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 528 FOLLOWER]: Advancing to term 529
I1014 19:27:15.299986 25435 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.300703 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 529 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.301017 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 529 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.301657 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 529 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.301678 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 529 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:15.301833 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:15.301894 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 529: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:15.302352 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 529 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 529: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:15.303812 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 529 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 529.
I1014 19:27:15.304180 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 529 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.304260 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 529 election: Election decided. Result: candidate won.
I1014 19:27:15.304683 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:15.304769 25435 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 FOLLOWER]: Leader election won for term 529
I1014 19:27:15.312350 25435 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:15.312721 25435 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 528.346, Last appended by leader: 346, Current term: 529, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.314132 25699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 529 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.314527 25699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.352216 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.352286 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:15.352337 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 529 FOLLOWER]: Advancing to term 530
I1014 19:27:15.354516 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.354979 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 530 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.355140 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 530 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.355754 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 530 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.355965 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 530 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.356381 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 530 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 530 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.367743 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 530 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.367910 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 529 FOLLOWER]: Advancing to term 530
I1014 19:27:15.367991 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:15.369814 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 530 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 530.
I1014 19:27:15.370252 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 530 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.370334 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 530 election: Election decided. Result: candidate won.
I1014 19:27:15.370589 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:15.370656 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 FOLLOWER]: Leader election won for term 530
I1014 19:27:15.370748 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:15.370962 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 528.346, Last appended by leader: 346, Current term: 530, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.371958 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 530 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.372165 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.374950 28660 generic_service.cc:146] {username='jenkins'} at 127.20.201.62:35095 changed flags via RPC: catalog_manager_inject_latency_load_ca_info_ms from '2000' to '0'
I1014 19:27:15.380532 28846 generic_service.cc:146] {username='jenkins'} at 127.20.201.61:57737 changed flags via RPC: catalog_manager_inject_latency_load_ca_info_ms from '2000' to '0'
I1014 19:27:15.387272 29031 generic_service.cc:146] {username='jenkins'} at 127.20.201.60:41616 changed flags via RPC: catalog_manager_inject_latency_load_ca_info_ms from '2000' to '0'
I1014 19:27:15.407054 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 529. Current term is 530. Ops: []
I1014 19:27:15.407644 25699 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.407846 25699 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 530.347 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 346
I1014 19:27:15.407953 25435 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 LEADER]: Stepping down as leader of term 529
I1014 19:27:15.408007 25435 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:15.408190 25435 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 529.347, Last appended by leader: 347, Current term: 529, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.408397 25435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:15.408450 25435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:15.408607 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 529 FOLLOWER]: Advancing to term 530
I1014 19:27:15.475039 25426 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 530 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.475136 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:15.475211 25426 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 530 FOLLOWER]: Advancing to term 531
I1014 19:27:15.477515 25426 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.477824 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 530. Current term is 531. Ops: []
I1014 19:27:15.478091 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 530 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 529 index: 347. Preceding OpId from leader: term: 530 index: 347. (term mismatch)
I1014 19:27:15.478144 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 346
I1014 19:27:15.478173 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 529.347
I1014 19:27:15.478215 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 530 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:15.478590 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 531 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.478776 25426 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 531 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.479223 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.479562 25651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.479643 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 531 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:15.479818 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 531 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.479948 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 531 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.480104 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 530 FOLLOWER]: Advancing to term 531
I1014 19:27:15.480170 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (vote granted)
I1014 19:27:15.480226 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 531 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 531 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.484035 25651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Stepping down as leader of term 530
I1014 19:27:15.484095 25651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:15.484287 25651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 530.347, Last appended by leader: 347, Current term: 530, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.484521 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:15.484557 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:15.484652 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 530 FOLLOWER]: Advancing to term 531
I1014 19:27:15.494999 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 531 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 531.
I1014 19:27:15.495198 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 531 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 530. Current term is 531. Ops: []
I1014 19:27:15.498569 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 531 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.498646 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 531 election: Election decided. Result: candidate won.
I1014 19:27:15.498900 25426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:15.498980 25426 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 FOLLOWER]: Leader election won for term 531
I1014 19:27:15.499128 25426 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:15.499377 25426 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 528.346, Last appended by leader: 346, Current term: 531, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.500687 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 531 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.500912 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.566135 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 531 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.566238 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:15.566294 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 531 FOLLOWER]: Advancing to term 532
I1014 19:27:15.568477 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.568972 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 532 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.569727 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 532 candidate_status { last_received { term: 530 index: 347 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.569929 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 532 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.570099 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 532 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.570443 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 532 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 532 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.570884 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 532 candidate_status { last_received { term: 530 index: 347 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.571023 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 531 FOLLOWER]: Advancing to term 532
I1014 19:27:15.571081 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:15.573104 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 532 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 532.
I1014 19:27:15.573632 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 532 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.573686 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 532 election: Election decided. Result: candidate won.
I1014 19:27:15.574019 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:15.574105 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 FOLLOWER]: Leader election won for term 532
I1014 19:27:15.574371 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:15.574618 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 530.347, Last appended by leader: 347, Current term: 532, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.576182 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 532 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.576373 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.586391 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 532 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 531. Current term is 532. Ops: []
I1014 19:27:15.587157 24617 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.587430 24617 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 LEADER]: Stepping down as leader of term 531
I1014 19:27:15.587486 24617 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:15.587687 24617 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 531.347, Last appended by leader: 347, Current term: 531, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.587869 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:15.587918 24617 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:15.588047 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 531 FOLLOWER]: Advancing to term 532
I1014 19:27:15.697530 25435 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 532 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.697612 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:15.697664 25435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 532 FOLLOWER]: Advancing to term 533
I1014 19:27:15.698204 24617 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 532 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.698282 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:15.698343 24617 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 532 FOLLOWER]: Advancing to term 533
I1014 19:27:15.699815 25435 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.700320 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 533 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.700503 25435 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 533 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.700472 24617 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.700939 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 533 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.701182 24617 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 533 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:15.701122 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 533 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:15.701184 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 533 candidate_status { last_received { term: 528 index: 346 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.701308 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 533 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.701400 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 533: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:15.701588 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 532. Current term is 533. Ops: []
I1014 19:27:15.701624 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 533 candidate_status { last_received { term: 531 index: 347 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.701779 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 533: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:15.702069 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 533 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 533 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:15.702674 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 533 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 533: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:15.702985 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 532. Current term is 533. Ops: []
I1014 19:27:15.703501 25651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 0.0, Next index: 348, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.703740 25651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 LEADER]: Stepping down as leader of term 532
I1014 19:27:15.703788 25651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:15.703974 25651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 532.348, Last appended by leader: 348, Current term: 532, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.704177 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:15.704226 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:15.704326 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 532 FOLLOWER]: Advancing to term 533
I1014 19:27:15.707370 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 533 candidate_status { last_received { term: 531 index: 347 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:15.707607 29010 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 533 because replica has last-logged OpId of term: 532 index: 348, which is greater than that of the candidate, which has last-logged OpId of term: 531 index: 347.
I1014 19:27:15.708009 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 533 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 533 because replica has last-logged OpId of term: 532 index: 348, which is greater than that of the candidate, which has last-logged OpId of term: 531 index: 347.
I1014 19:27:15.708104 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 533 election: Election decided. Result: candidate lost.
I1014 19:27:15.708093 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 533 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 533: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:15.708166 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 533 election: Election decided. Result: candidate lost.
I1014 19:27:15.708379 25435 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:15.708448 25435 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Leader election lost for term 533. Reason: could not achieve majority
I1014 19:27:15.708570 24617 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (election complete)
I1014 19:27:15.708667 24617 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Leader election lost for term 533. Reason: could not achieve majority
W1014 19:27:15.757416 25298 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Initializing Kudu internal certificate authority: real 1.476s	user 0.000s	sys 0.000s
I1014 19:27:15.757467 25298 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:15.758152 25298 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:15.758955 25298 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 524 vs 533: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 533 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:15.759135 25298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 533 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:15.759280 25298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 533 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:15.759405 25298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 533 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
W1014 19:27:15.759577 25298 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 533 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.787955 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 533 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:15.788043 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:15.788105 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 533 FOLLOWER]: Advancing to term 534
I1014 19:27:15.790268 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.790839 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 534 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.790999 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 534 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.791612 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 534 candidate_status { last_received { term: 532 index: 348 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:15.791759 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 533 FOLLOWER]: Advancing to term 534
I1014 19:27:15.791831 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:15.793802 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 534 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 534.
I1014 19:27:15.795207 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 534 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:15.795274 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 534 election: Election decided. Result: candidate won.
I1014 19:27:15.795411 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 534 candidate_status { last_received { term: 532 index: 348 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:15.795605 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 533 FOLLOWER]: Advancing to term 534
I1014 19:27:15.795687 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.194s (vote granted)
I1014 19:27:15.797801 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 534 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 534.
I1014 19:27:15.798199 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 534 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:15.803547 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:15.803740 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 FOLLOWER]: Leader election won for term 534
I1014 19:27:15.803869 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:15.804123 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 532.348, Last appended by leader: 348, Current term: 534, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:15.807021 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.807272 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.900028 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 534 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 528 index: 346. Preceding OpId from leader: term: 534 index: 349. (index mismatch)
I1014 19:27:15.900712 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.901996 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.902132 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.915014 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.915279 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:15.915771 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 534 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 531 index: 347. Preceding OpId from leader: term: 534 index: 349. (index mismatch)
I1014 19:27:15.916244 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 346, Time since last communication: 0.000s
I1014 19:27:15.916416 25432 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer 88e69d68a7914bf48d802a2f2e500b71 log is divergent from this leader: its last log entry 531.347 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 346
I1014 19:27:15.917250 28639 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting all transactions after (but not including) 346
I1014 19:27:15.917292 28639 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Aborting uncommitted NO_OP operation due to leader change: 531.347
I1014 19:27:15.917348 28639 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 534 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:15.926168 25435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.926162 25849 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.926378 25849 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.926378 25435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.927073 24617 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.927096 25853 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.927361 25853 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.927361 24617 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.930233 25426 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.930421 25426 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.930583 25851 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.930757 25851 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:15.930914 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:15.931053 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:15.999075 21475 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Initializing Kudu internal certificate authority: real 1.277s	user 0.001s	sys 0.000s
I1014 19:27:15.999122 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.000084 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:16.002388 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 534 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.002643 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.006551 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:16.009677 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:16.013128 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.016413 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:16.206935 25851 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 534 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:16.207042 25851 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:16.207118 25851 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 534 FOLLOWER]: Advancing to term 535
I1014 19:27:16.209333 25851 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 535 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.209846 25851 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 535 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.210064 25851 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 535 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.210536 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 535 candidate_status { last_received { term: 534 index: 351 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.210721 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 534 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 535 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.210635 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 535 candidate_status { last_received { term: 534 index: 351 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.210830 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 535 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.213918 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 535 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 534 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 535 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.214615 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 535 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 535 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.214687 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 535 election: Election decided. Result: candidate lost.
I1014 19:27:16.214915 25851 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:16.214984 25851 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 535 FOLLOWER]: Leader election lost for term 535. Reason: could not achieve majority
I1014 19:27:16.229161 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 535 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 534. Current term is 535. Ops: []
I1014 19:27:16.229826 25432 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 534.351, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I1014 19:27:16.230069 25651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 LEADER]: Stepping down as leader of term 534
I1014 19:27:16.230114 25651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.230320 25651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 534.351, Last appended by leader: 351, Current term: 534, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.230515 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:16.230545 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:16.230629 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 534 FOLLOWER]: Advancing to term 535
I1014 19:27:16.306385 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 535 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:16.306458 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:16.306537 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 535 FOLLOWER]: Advancing to term 536
I1014 19:27:16.308672 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.309161 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 536 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.309329 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 536 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.309931 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 536 candidate_status { last_received { term: 534 index: 351 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:16.310092 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 535 FOLLOWER]: Advancing to term 536
I1014 19:27:16.310176 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (vote granted)
I1014 19:27:16.312383 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 536 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 536.
I1014 19:27:16.312983 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 536 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.313038 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 536 election: Election decided. Result: candidate won.
I1014 19:27:16.313258 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:16.313336 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 FOLLOWER]: Leader election won for term 536
I1014 19:27:16.313457 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.313664 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 351, Committed index: 351, Last appended: 534.351, Last appended by leader: 351, Current term: 536, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.314230 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 536 candidate_status { last_received { term: 534 index: 351 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.314390 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 534 FOLLOWER]: Advancing to term 536
I1014 19:27:16.314466 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:16.316535 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 536.
I1014 19:27:16.316534 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.316740 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.316934 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:16.317065 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 536 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.318804 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:16.320135 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.321432 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:16.324246 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 536 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 534 index: 351. Preceding OpId from leader: term: 536 index: 353. (index mismatch)
I1014 19:27:16.324564 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 534 index: 351. Preceding OpId from leader: term: 536 index: 353. (index mismatch)
I1014 19:27:16.325027 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I1014 19:27:16.325549 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 352, Last known committed idx: 351, Time since last communication: 0.000s
I1014 19:27:16.326731 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.326951 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.328021 25851 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.328205 25851 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.335008 26013 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.335546 25851 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.335739 25851 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.336511 26013 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.344146 26012 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.344439 26012 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.347148 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 536 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.347342 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.435320 25851 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 536 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:16.435438 25851 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:16.435514 25851 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 536 FOLLOWER]: Advancing to term 537
I1014 19:27:16.437661 25851 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.438168 25851 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 537 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.439276 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 537 candidate_status { last_received { term: 536 index: 353 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.439360 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 536. Current term is 537. Ops: []
I1014 19:27:16.439498 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 537 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.439630 25851 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 537 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.439891 25432 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 536.353, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I1014 19:27:16.440165 25432 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 LEADER]: Stepping down as leader of term 536
I1014 19:27:16.440208 25432 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.440376 25432 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 536.353, Last appended by leader: 353, Current term: 536, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.440604 25432 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:16.440639 25432 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:16.440759 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 536 FOLLOWER]: Advancing to term 537
I1014 19:27:16.439945 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 537 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 537 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.444047 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 537 candidate_status { last_received { term: 536 index: 353 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.444200 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (vote granted)
I1014 19:27:16.446930 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 537.
I1014 19:27:16.447302 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 537 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.447391 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 537 election: Election decided. Result: candidate won.
I1014 19:27:16.450559 25852 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:16.450644 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:16.450700 25852 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 536 FOLLOWER]: Advancing to term 537
I1014 19:27:16.452852 25852 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 537 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.453335 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 537 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.453512 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 537 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.454324 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 537 candidate_status { last_received { term: 536 index: 353 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.454547 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 537: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.454978 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 537 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 537: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.459843 25851 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:16.459929 25851 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 FOLLOWER]: Leader election won for term 537
I1014 19:27:16.460863 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 537 candidate_status { last_received { term: 536 index: 353 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
W1014 19:27:16.468986 25695 catalog_manager.cc:1038] Time spent T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Initializing Kudu internal certificate authority: real 1.451s	user 0.001s	sys 0.000s
I1014 19:27:16.469039 25695 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.469704 25851 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:16.469938 25695 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:16.470255 25851 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 536.353, Last appended by leader: 353, Current term: 537, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.471925 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 537 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.472249 26033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.472506 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 537 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 537 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.472579 26033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.472594 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 537 election: Election decided. Result: candidate lost.
I1014 19:27:16.472926 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:16.472988 25852 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 537 FOLLOWER]: Leader election lost for term 537. Reason: could not achieve majority
I1014 19:27:16.474231 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 536 index: 353. Preceding OpId from leader: term: 537 index: 355. (index mismatch)
I1014 19:27:16.474742 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 537 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 536 index: 353. Preceding OpId from leader: term: 537 index: 355. (index mismatch)
I1014 19:27:16.475394 26033 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I1014 19:27:16.476182 26033 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I1014 19:27:16.477164 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.479089 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.479259 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.481571 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.481767 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.502727 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.508373 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.508601 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.514405 25695 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:16.514396 25851 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.514641 25851 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.514528 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 537 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.514792 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.516317 25695 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:16.519356 25695 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.527966 25695 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:16.638459 25852 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 537 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:16.638552 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:16.638605 25852 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 537 FOLLOWER]: Advancing to term 538
I1014 19:27:16.640681 25852 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.641166 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 538 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.641324 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 538 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.641824 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 538 candidate_status { last_received { term: 537 index: 356 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:16.642031 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 538 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.642062 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 538 candidate_status { last_received { term: 537 index: 356 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.642247 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 538 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.642587 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 538 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 538 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.642760 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 538 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 538 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.642824 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 538 election: Election decided. Result: candidate lost.
I1014 19:27:16.643200 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:16.643259 25852 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Leader election lost for term 538. Reason: could not achieve majority
I1014 19:27:16.652297 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 537. Current term is 538. Ops: []
I1014 19:27:16.652766 26034 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 537.356, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I1014 19:27:16.652982 26034 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Stepping down as leader of term 537
I1014 19:27:16.653017 26034 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:16.653170 26034 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 537.356, Last appended by leader: 356, Current term: 537, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.653342 26034 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:16.653375 26034 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:16.653501 26034 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 537 FOLLOWER]: Advancing to term 538
I1014 19:27:16.725258 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:16.725333 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:16.725384 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 537 FOLLOWER]: Advancing to term 538
I1014 19:27:16.727494 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.727967 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 538 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.728109 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 538 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.731081 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 538 candidate_status { last_received { term: 537 index: 356 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.731258 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 538: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:16.731670 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 538 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 538: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:16.735618 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 538 candidate_status { last_received { term: 537 index: 356 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:16.735759 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:16.737766 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 538 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 538.
I1014 19:27:16.739325 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 538 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.739392 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 538 election: Election decided. Result: candidate won.
I1014 19:27:16.739609 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:16.739673 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 FOLLOWER]: Leader election won for term 538
I1014 19:27:16.739764 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.740037 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 537.356, Last appended by leader: 356, Current term: 538, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.740998 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.741238 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.741432 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:16.743104 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:16.744352 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.746237 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:16.749290 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 537 index: 356. Preceding OpId from leader: term: 538 index: 358. (index mismatch)
I1014 19:27:16.749586 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 538 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 537 index: 356. Preceding OpId from leader: term: 538 index: 358. (index mismatch)
I1014 19:27:16.750035 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.002s
I1014 19:27:16.752336 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I1014 19:27:16.756880 26013 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.757138 26013 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.757369 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.757586 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.757526 26033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.757707 26033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.759027 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.759202 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.759676 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.759843 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.760839 26033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 538 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.760984 26033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:16.859506 26033 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 538 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:16.859578 26033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:16.859625 26033 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 538 FOLLOWER]: Advancing to term 539
I1014 19:27:16.860647 25852 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:16.860713 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:16.860765 25852 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 538 FOLLOWER]: Advancing to term 539
I1014 19:27:16.861709 26033 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.862212 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 539 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.862350 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 539 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.862751 25852 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.863050 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 539 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.863227 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 539: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:16.863261 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 539 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.863400 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 539 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.863912 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 539 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.864094 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 539 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.864411 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 539 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:16.864619 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 539: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.864650 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 539 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.864861 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 539 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.865067 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 538. Current term is 539. Ops: []
I1014 19:27:16.865562 26014 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 538.358, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I1014 19:27:16.865849 26014 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Stepping down as leader of term 538
I1014 19:27:16.865895 26014 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.866103 26014 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 538.358, Last appended by leader: 358, Current term: 538, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.866333 26014 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:16.866375 26014 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:16.866492 26014 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 FOLLOWER]: Advancing to term 539
I1014 19:27:16.869096 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 539 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 539: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:16.869316 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 539 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 539 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.869545 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 539 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 538 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 539 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:16.869611 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 539 election: Election decided. Result: candidate lost.
I1014 19:27:16.869813 26033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:16.869879 26033 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Leader election lost for term 539. Reason: could not achieve majority
I1014 19:27:16.870182 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 539 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 539: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.870246 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 539 election: Election decided. Result: candidate lost.
I1014 19:27:16.870290 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 538. Current term is 539. Ops: []
I1014 19:27:16.870646 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:16.870729 25852 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Leader election lost for term 539. Reason: could not achieve majority
I1014 19:27:16.981009 26033 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:16.981086 26033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.215s (starting election)
I1014 19:27:16.981142 26033 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 539 FOLLOWER]: Advancing to term 540
I1014 19:27:16.981250 25651 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 539 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:16.981314 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:16.981359 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 539 FOLLOWER]: Advancing to term 540
I1014 19:27:16.983285 26033 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.983398 25651 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.983814 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 540 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.983902 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 540 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:16.984086 25651 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 540 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.985003 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 540 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:16.985872 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 540 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:16.986042 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 540: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.986193 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 540 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.986352 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 539 FOLLOWER]: Advancing to term 540
I1014 19:27:16.986424 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.195s (vote granted)
I1014 19:27:16.986542 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 540 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:16.986706 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 540: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:16.986851 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 540 candidate_status { last_received { term: 538 index: 358 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:16.988592 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 540.
I1014 19:27:16.989362 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 540 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:16.989429 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 540 election: Election decided. Result: candidate won.
I1014 19:27:16.989832 25651 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (election complete)
I1014 19:27:16.989903 25651 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 FOLLOWER]: Leader election won for term 540
I1014 19:27:16.990083 25651 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:16.990285 25651 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 538.358, Last appended by leader: 358, Current term: 540, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:16.990556 28826 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 540 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:16.991128 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 540 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 540: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:16.991374 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 540 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 540: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:16.991394 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 540 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 540 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:16.991452 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 540 election: Election decided. Result: candidate lost.
I1014 19:27:16.993326 26033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.193s (election complete)
I1014 19:27:16.993388 26033 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Leader election lost for term 540. Reason: could not achieve majority
I1014 19:27:16.993518 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:16.993716 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:16.993940 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:16.995838 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:16.997190 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:16.999094 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:17.001454 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 538 index: 358. Preceding OpId from leader: term: 540 index: 360. (index mismatch)
I1014 19:27:17.002149 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 538 index: 358. Preceding OpId from leader: term: 540 index: 360. (index mismatch)
I1014 19:27:17.002423 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I1014 19:27:17.003726 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I1014 19:27:17.004555 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.004776 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.005857 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.006047 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.006994 26033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.007213 26033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.007838 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.008038 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.008338 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.008545 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.010771 26033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 540 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.010900 26033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.110570 26033 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.110646 26033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:17.110702 26033 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 540 FOLLOWER]: Advancing to term 541
I1014 19:27:17.112774 26033 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.113296 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 541 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.113441 26033 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 541 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.114076 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 541 candidate_status { last_received { term: 540 index: 360 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.114254 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 541 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.114464 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 541 candidate_status { last_received { term: 540 index: 360 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.114703 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 541 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.115149 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 541 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 541 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.115460 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 541 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 541 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.115567 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 541 election: Election decided. Result: candidate lost.
I1014 19:27:17.115783 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:17.115850 26034 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Leader election lost for term 541. Reason: could not achieve majority
I1014 19:27:17.120657 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 540. Current term is 541. Ops: []
I1014 19:27:17.121196 25651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 540.360, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I1014 19:27:17.121456 25651 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 LEADER]: Stepping down as leader of term 540
I1014 19:27:17.121533 25651 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:17.121897 25651 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 540.360, Last appended by leader: 360, Current term: 540, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.122088 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:17.122124 25651 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:17.122231 25651 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 540 FOLLOWER]: Advancing to term 541
I1014 19:27:17.187650 25852 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.187724 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:17.187772 25852 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 540 FOLLOWER]: Advancing to term 541
I1014 19:27:17.189828 25852 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.190295 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 541 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.190492 25852 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 541 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.191146 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 541 candidate_status { last_received { term: 540 index: 360 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:17.191363 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 541: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.191785 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 541 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 541: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.191840 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 541 candidate_status { last_received { term: 540 index: 360 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.192016 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:27:17.194023 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 541.
I1014 19:27:17.194372 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 541 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.194433 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 541 election: Election decided. Result: candidate won.
I1014 19:27:17.197888 25852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:17.197966 25852 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 FOLLOWER]: Leader election won for term 541
I1014 19:27:17.198453 25852 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:17.198698 25852 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 360, Committed index: 360, Last appended: 540.360, Last appended by leader: 360, Current term: 541, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.207648 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.207983 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.212939 26205 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:17.215412 26205 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:17.216785 26205 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:17.217662 26205 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:17.220288 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 540 index: 360. Preceding OpId from leader: term: 541 index: 362. (index mismatch)
I1014 19:27:17.220840 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 540 index: 360. Preceding OpId from leader: term: 541 index: 362. (index mismatch)
I1014 19:27:17.220810 25852 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I1014 19:27:17.221513 26204 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 361, Last known committed idx: 360, Time since last communication: 0.000s
I1014 19:27:17.222386 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.222600 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.223325 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.223533 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.228034 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.229176 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.229094 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.229305 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.229701 25847 mvcc.cc:203] Tried to move safe_time back from 6306080510847569920 to 6306080510770552832. Current Snapshot: MvccSnapshot[committed={T|T < 6306080510847569920 or (T in {6306080510847569920})}]
I1014 19:27:17.240046 25852 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.240252 25852 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.240962 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 541 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.241137 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.341814 26034 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:17.341890 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (starting election)
I1014 19:27:17.341938 26034 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 541 FOLLOWER]: Advancing to term 542
I1014 19:27:17.343992 26034 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.344518 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 542 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.344686 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 542 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.345206 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 542 candidate_status { last_received { term: 541 index: 362 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.345396 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 542 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.345392 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 542 candidate_status { last_received { term: 541 index: 362 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.345613 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 542 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.346004 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 542 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 542 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.346421 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 542 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 542 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.346509 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 542 election: Election decided. Result: candidate lost.
I1014 19:27:17.346695 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (election complete)
I1014 19:27:17.346751 26034 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Leader election lost for term 542. Reason: could not achieve majority
I1014 19:27:17.353495 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 541. Current term is 542. Ops: []
I1014 19:27:17.354058 26207 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 541.362, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I1014 19:27:17.354317 26204 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 LEADER]: Stepping down as leader of term 541
I1014 19:27:17.354367 26204 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:17.354578 26204 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 541.362, Last appended by leader: 362, Current term: 541, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.354755 26204 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:17.354787 26204 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:17.354889 26204 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 541 FOLLOWER]: Advancing to term 542
I1014 19:27:17.420172 25432 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:17.420246 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:17.420303 25432 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 541 FOLLOWER]: Advancing to term 542
I1014 19:27:17.422493 25432 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.423051 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 542 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.423228 25432 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 542 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.424046 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 542 candidate_status { last_received { term: 541 index: 362 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:17.424244 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 542: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.424551 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 542 candidate_status { last_received { term: 541 index: 362 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.424708 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (vote granted)
I1014 19:27:17.426749 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 542.
I1014 19:27:17.427418 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 542 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.427481 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 542 election: Election decided. Result: candidate won.
I1014 19:27:17.427765 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 542 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 542: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.428340 25432 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:17.428437 25432 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 FOLLOWER]: Leader election won for term 542
I1014 19:27:17.430188 25432 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:17.430441 25432 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 541.362, Last appended by leader: 362, Current term: 542, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.431743 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.431955 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.432163 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:17.433768 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:17.434903 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:17.435814 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:17.439956 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 541 index: 362. Preceding OpId from leader: term: 542 index: 364. (index mismatch)
I1014 19:27:17.440784 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 541 index: 362. Preceding OpId from leader: term: 542 index: 364. (index mismatch)
I1014 19:27:17.441344 26014 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I1014 19:27:17.441722 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I1014 19:27:17.443497 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.443691 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.444707 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.444900 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.444900 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.445093 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.445686 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.445880 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.446346 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.446549 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.447384 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 542 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.447584 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.609045 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:17.625334 26034 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.625409 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:17.625460 26034 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 542 FOLLOWER]: Advancing to term 543
I1014 19:27:17.627542 26034 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.628021 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 543 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.628168 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 543 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.629083 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 543 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.629262 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 543 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.629277 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 543 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.629457 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 543 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.629874 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 543 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 543 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.630235 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 543 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 543 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.630295 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 543 election: Election decided. Result: candidate lost.
I1014 19:27:17.630667 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:17.630723 26034 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Leader election lost for term 543. Reason: could not achieve majority
I1014 19:27:17.646387 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 542. Current term is 543. Ops: []
I1014 19:27:17.647425 25432 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 542.364, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I1014 19:27:17.647711 26014 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 LEADER]: Stepping down as leader of term 542
I1014 19:27:17.647759 26014 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:17.647948 26014 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 542.364, Last appended by leader: 364, Current term: 542, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.648144 26014 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:17.648188 26014 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:17.648329 26014 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 542 FOLLOWER]: Advancing to term 543
I1014 19:27:17.723408 26204 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.723512 26204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:17.723569 26204 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 542 FOLLOWER]: Advancing to term 543
I1014 19:27:17.725118 26204 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 543 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.725515 26204 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 543 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.725656 26204 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 543 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.726117 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 543 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:17.726315 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 543: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.726856 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 543 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 543: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.727905 26014 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 543 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:17.727970 26014 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:17.728024 26014 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 543 FOLLOWER]: Advancing to term 544
I1014 19:27:17.728188 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 543 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.730041 26014 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.730501 26014 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 544 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.730670 26014 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 544 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.730737 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 543. Current term is 544.
W1014 19:27:17.731144 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 543 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 543. Current term is 544.
I1014 19:27:17.731184 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 543 election: Cancelling election due to peer responding with higher term
I1014 19:27:17.731123 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 544 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:17.731250 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 543 FOLLOWER]: Advancing to term 544
I1014 19:27:17.731320 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (vote granted)
I1014 19:27:17.731362 26204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:17.731421 26204 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 543 FOLLOWER]: Advancing to term 544
I1014 19:27:17.732755 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 544 candidate_status { last_received { term: 542 index: 364 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.733311 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 544 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 544.
I1014 19:27:17.733784 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 544 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.733839 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 544 election: Election decided. Result: candidate won.
I1014 19:27:17.734027 26014 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:17.734092 26014 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Leader election won for term 544
I1014 19:27:17.734171 26014 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:17.734362 26014 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 542.364, Last appended by leader: 364, Current term: 544, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.735954 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.736124 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.736291 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:17.738143 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:17.743438 26204 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Leader election lost for term 543. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 543. Current term is 544.
I1014 19:27:17.743587 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (vote granted)
I1014 19:27:17.743815 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:17.744752 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:17.745597 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 544.
I1014 19:27:17.746130 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 544 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.747506 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 544 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 542 index: 364. Preceding OpId from leader: term: 544 index: 366. (index mismatch)
I1014 19:27:17.747983 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 542 index: 364. Preceding OpId from leader: term: 544 index: 366. (index mismatch)
I1014 19:27:17.748005 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I1014 19:27:17.748970 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I1014 19:27:17.751088 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.751320 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.751248 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.751423 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.752741 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.752946 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.752984 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.753161 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.755439 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.756335 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.757268 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 544 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.757441 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.856426 26034 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 544 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.856523 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:17.856575 26034 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 544 FOLLOWER]: Advancing to term 545
I1014 19:27:17.858669 26034 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.859203 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 545 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:17.859350 26034 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 545 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.860687 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 545 candidate_status { last_received { term: 544 index: 366 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:17.860862 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 545 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.861027 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 545 candidate_status { last_received { term: 544 index: 366 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.861230 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 545 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.861224 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 545 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 545 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.861809 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 545 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 545 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:17.861869 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 545 election: Election decided. Result: candidate lost.
I1014 19:27:17.862123 26034 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:17.862187 26034 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Leader election lost for term 545. Reason: could not achieve majority
I1014 19:27:17.869158 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 544. Current term is 545. Ops: []
I1014 19:27:17.869658 25651 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 544.366, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I1014 19:27:17.869911 26302 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 LEADER]: Stepping down as leader of term 544
I1014 19:27:17.869967 26302 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:17.870234 26302 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 544.366, Last appended by leader: 366, Current term: 544, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.870502 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:17.870543 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:17.870659 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 544 FOLLOWER]: Advancing to term 545
I1014 19:27:17.938807 26204 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:17.938881 26204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:17.938947 26204 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 544 FOLLOWER]: Advancing to term 545
I1014 19:27:17.941081 26204 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.941601 26204 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 545 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:17.941720 26204 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 545 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.942170 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 545 candidate_status { last_received { term: 544 index: 366 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:17.942375 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 545: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.942728 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 545 candidate_status { last_received { term: 544 index: 366 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:17.942886 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (vote granted)
I1014 19:27:17.944953 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 545 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 545.
I1014 19:27:17.946506 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 545 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:17.946597 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 545 election: Election decided. Result: candidate won.
I1014 19:27:17.946838 26204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:17.946913 26204 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 FOLLOWER]: Leader election won for term 545
I1014 19:27:17.946882 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 545 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 545: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:17.947306 26204 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:17.947525 26204 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 544.366, Last appended by leader: 366, Current term: 545, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:17.948808 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.949132 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.954391 26392 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:17.956934 26392 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:17.959177 26392 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:17.960090 26392 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:17.962429 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 544 index: 366. Preceding OpId from leader: term: 545 index: 368. (index mismatch)
I1014 19:27:17.963027 26204 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I1014 19:27:17.963690 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 545 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 544 index: 366. Preceding OpId from leader: term: 545 index: 368. (index mismatch)
I1014 19:27:17.964702 26204 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I1014 19:27:17.966018 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.966233 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.967303 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.967537 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.969755 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.970012 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.970149 26204 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.970306 26204 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:17.970885 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.971071 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:17.973336 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 545 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:17.973577 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.072134 26302 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 545 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:18.072217 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:18.072278 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 545 FOLLOWER]: Advancing to term 546
I1014 19:27:18.073958 26302 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.074369 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 546 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.074518 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 546 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.075142 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 546 candidate_status { last_received { term: 545 index: 368 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.075366 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 546 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.075615 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 545. Current term is 546. Ops: []
I1014 19:27:18.075754 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 546 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 546 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.076581 26390 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 545.368, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I1014 19:27:18.076818 26390 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 LEADER]: Stepping down as leader of term 545
I1014 19:27:18.076861 26390 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:18.077040 26390 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 545.368, Last appended by leader: 368, Current term: 545, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.077229 26390 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:18.077266 26390 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:18.077404 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 545 FOLLOWER]: Advancing to term 546
I1014 19:27:18.078629 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 546 candidate_status { last_received { term: 545 index: 368 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.079711 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (vote granted)
I1014 19:27:18.081681 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 546 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 546.
I1014 19:27:18.082144 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 546 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.082221 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 546 election: Election decided. Result: candidate won.
I1014 19:27:18.082438 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:18.082532 26302 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 FOLLOWER]: Leader election won for term 546
I1014 19:27:18.082618 26302 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:18.082809 26302 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 545.368, Last appended by leader: 368, Current term: 546, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.084192 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.084368 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.084568 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:18.087020 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:18.088393 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:18.095414 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:18.099045 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 546 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 545 index: 368. Preceding OpId from leader: term: 546 index: 370. (index mismatch)
I1014 19:27:18.099107 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 545 FOLLOWER]: Advancing to term 546
I1014 19:27:18.099622 25651 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I1014 19:27:18.101374 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 546 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 545 index: 368. Preceding OpId from leader: term: 546 index: 370. (index mismatch)
I1014 19:27:18.101917 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.102115 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.102453 26014 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I1014 19:27:18.103188 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.103370 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.104329 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.104537 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.129331 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.129542 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.129247 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.129655 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.180652 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 546 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.180871 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.204300 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 546 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:18.204382 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:18.204463 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 546 FOLLOWER]: Advancing to term 547
I1014 19:27:18.206488 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.207012 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 547 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.207199 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 547 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.209647 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 547 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.209856 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 546 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 547 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.209911 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 547 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.210094 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 547 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.210660 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 547 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 547 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.210834 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 547 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 546 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 547 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.210911 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 547 election: Election decided. Result: candidate lost.
I1014 19:27:18.211328 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:18.211413 26390 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Leader election lost for term 547. Reason: could not achieve majority
I1014 19:27:18.214844 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 546. Current term is 547. Ops: []
I1014 19:27:18.215443 25432 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 546.370, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I1014 19:27:18.215708 26302 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 LEADER]: Stepping down as leader of term 546
I1014 19:27:18.215752 26302 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:18.215924 26302 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 546.370, Last appended by leader: 370, Current term: 546, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.216112 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:18.216150 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:18.216290 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 546 FOLLOWER]: Advancing to term 547
I1014 19:27:18.282429 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 546 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:18.282510 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:18.282564 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 546 FOLLOWER]: Advancing to term 547
I1014 19:27:18.284787 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 547 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.285290 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 547 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.298863 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 547 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.299078 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 547: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.299111 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 547 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.299500 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 547 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 547: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.299579 26302 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 547 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:18.299660 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:18.299720 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 547 FOLLOWER]: Advancing to term 548
I1014 19:27:18.300179 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 547 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.301868 26302 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.302343 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 548 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.302621 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 548 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.302893 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 548 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.303036 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 547 FOLLOWER]: Advancing to term 548
I1014 19:27:18.303088 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (vote granted)
I1014 19:27:18.303243 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 548 candidate_status { last_received { term: 546 index: 370 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.303386 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 547 FOLLOWER]: Advancing to term 548
I1014 19:27:18.303452 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (vote granted)
I1014 19:27:18.304939 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 548.
I1014 19:27:18.305552 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 548 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 548.
I1014 19:27:18.305656 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 547. Current term is 548.
I1014 19:27:18.306067 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 548 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.306143 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 548 election: Election decided. Result: candidate won.
I1014 19:27:18.306401 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:18.306532 26302 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Leader election won for term 548
W1014 19:27:18.306547 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 547 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 547. Current term is 548.
I1014 19:27:18.306576 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 547 election: Cancelling election due to peer responding with higher term
I1014 19:27:18.306768 26302 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:18.306790 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:18.306838 26439 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Leader election lost for term 547. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 547. Current term is 548.
I1014 19:27:18.306993 26302 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 546.370, Last appended by leader: 370, Current term: 548, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.307579 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 548 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.308399 25432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.308619 25432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.308840 21475 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:18.310969 21475 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:18.312374 21475 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:18.313875 21475 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:18.316453 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 548 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 546 index: 370. Preceding OpId from leader: term: 548 index: 372. (index mismatch)
I1014 19:27:18.316627 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 546 index: 370. Preceding OpId from leader: term: 548 index: 372. (index mismatch)
I1014 19:27:18.317167 25432 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I1014 19:27:18.318877 26014 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I1014 19:27:18.319074 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.319263 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.321390 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.321605 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.321693 26014 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.321882 26014 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.322227 25651 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.322407 25651 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.323606 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.323593 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 548 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.323781 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.323783 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.423118 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 548 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:18.423194 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:18.423259 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 548 FOLLOWER]: Advancing to term 549
I1014 19:27:18.425397 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.425936 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 549 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.426102 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 549 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.426738 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 549 candidate_status { last_received { term: 548 index: 372 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.426923 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 549 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.427220 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 549 candidate_status { last_received { term: 548 index: 372 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.427409 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 549 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.427783 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 549 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 549 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.427918 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 549 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 549 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.427984 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 549 election: Election decided. Result: candidate lost.
I1014 19:27:18.428187 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:18.428251 26390 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Leader election lost for term 549. Reason: could not achieve majority
I1014 19:27:18.431924 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 548. Current term is 549. Ops: []
I1014 19:27:18.432435 26302 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 548.372, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I1014 19:27:18.432765 26302 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 LEADER]: Stepping down as leader of term 548
I1014 19:27:18.432814 26302 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:18.433007 26302 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 548.372, Last appended by leader: 372, Current term: 548, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.433182 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:18.433221 26302 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:18.433364 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 548 FOLLOWER]: Advancing to term 549
I1014 19:27:18.511190 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:18.511263 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:18.511314 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 548 FOLLOWER]: Advancing to term 549
I1014 19:27:18.513379 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.513864 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 549 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.514061 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 549 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.514746 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 549 candidate_status { last_received { term: 548 index: 372 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.514746 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 549 candidate_status { last_received { term: 548 index: 372 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.514927 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:18.514963 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 549: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.515415 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 549 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 549: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.517129 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 549 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 549.
I1014 19:27:18.517570 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 549 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.517642 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 549 election: Election decided. Result: candidate won.
I1014 19:27:18.517861 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:18.517925 26439 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 FOLLOWER]: Leader election won for term 549
I1014 19:27:18.518033 26439 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:18.518242 26439 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 548.372, Last appended by leader: 372, Current term: 549, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.520431 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.520632 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.524641 26511 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:18.526537 26511 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:18.527856 26511 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:18.528738 26511 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:18.531430 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 548 index: 372. Preceding OpId from leader: term: 549 index: 374. (index mismatch)
I1014 19:27:18.531497 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 549 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 548 index: 372. Preceding OpId from leader: term: 549 index: 374. (index mismatch)
I1014 19:27:18.532029 26034 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I1014 19:27:18.532229 26439 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I1014 19:27:18.533911 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.534132 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.534102 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.534309 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.536944 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.537487 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.538714 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.538993 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.539250 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.539479 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.539680 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 549 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.540238 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.587684 29304 heartbeater.cc:467] Master 127.20.201.60:53788 was elected leader, sending a full tablet report...
I1014 19:27:18.638700 26302 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 549 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:18.638782 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:18.638860 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 549 FOLLOWER]: Advancing to term 550
I1014 19:27:18.641002 26302 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.641490 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 550 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.641664 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 550 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.642375 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:18.642442 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:18.642490 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 549 FOLLOWER]: Advancing to term 550
I1014 19:27:18.642935 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 550 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.643126 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 550 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.643589 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 550 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 550 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.644552 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 550 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.644978 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 550 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.645102 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 550 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.645329 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 550 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.645478 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 550 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 550: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.645942 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 550 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 550 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 550: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:18.646008 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 550 election: Election decided. Result: candidate lost.
I1014 19:27:18.646309 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 550 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.646471 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 550 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.646400 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 550 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.646591 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 550: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.646884 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 550 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 550 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.647078 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 550 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 550: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.647130 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 550 election: Election decided. Result: candidate lost.
I1014 19:27:18.647403 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:18.647454 26390 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 550 FOLLOWER]: Leader election lost for term 550. Reason: could not achieve majority
I1014 19:27:18.647593 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:18.647642 26302 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Leader election lost for term 550. Reason: could not achieve majority
I1014 19:27:18.653903 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 549. Current term is 550. Ops: []
I1014 19:27:18.654393 26439 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 549.374, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I1014 19:27:18.654651 26512 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Stepping down as leader of term 549
I1014 19:27:18.654705 26512 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:18.654901 26512 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 549.374, Last appended by leader: 374, Current term: 549, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.655141 26512 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:18.655189 26512 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:18.655325 26512 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 549 FOLLOWER]: Advancing to term 550
I1014 19:27:18.749419 26512 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 550 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:18.749500 26512 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:18.749564 26512 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 550 FOLLOWER]: Advancing to term 551
I1014 19:27:18.751377 26302 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:18.751454 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.158s (starting election)
I1014 19:27:18.751533 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 550 FOLLOWER]: Advancing to term 551
I1014 19:27:18.751698 26512 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.752176 26512 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 551 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.752316 26512 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 551 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.752880 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 551 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.753027 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 550 FOLLOWER]: Advancing to term 551
I1014 19:27:18.753085 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.162s (vote granted)
I1014 19:27:18.753182 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 551 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.753480 26302 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.753813 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 551: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.753945 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 551 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.754077 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 551 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.754220 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 551 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 551: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.755157 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 551.
I1014 19:27:18.755566 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 551 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.755617 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 551 election: Election decided. Result: candidate won.
I1014 19:27:18.756243 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 551 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.756450 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 551: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:18.757992 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 551 candidate_status { last_received { term: 549 index: 374 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.758152 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 551: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:18.758554 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 551 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 551: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:18.759603 26512 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:18.759707 26512 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 FOLLOWER]: Leader election won for term 551
I1014 19:27:18.759930 26512 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:18.760195 26512 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 549.374, Last appended by leader: 374, Current term: 551, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.761732 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.761899 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.762301 26511 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:18.764235 26511 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:18.767076 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 551 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 551: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:18.767179 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 551 election: Election decided. Result: candidate lost.
I1014 19:27:18.768292 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.157s (election complete)
I1014 19:27:18.768371 26302 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Leader election lost for term 551. Reason: could not achieve majority
I1014 19:27:18.768630 26511 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:18.769624 26511 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:18.772996 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 549 index: 374. Preceding OpId from leader: term: 551 index: 376. (index mismatch)
I1014 19:27:18.773157 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 549 index: 374. Preceding OpId from leader: term: 551 index: 376. (index mismatch)
I1014 19:27:18.773560 26439 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I1014 19:27:18.775264 26439 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I1014 19:27:18.777020 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.777232 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.778213 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.778407 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.779721 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.779901 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.779847 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.780081 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.780846 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.781021 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.783838 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 551 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.784101 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:18.880136 26302 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:18.880213 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:18.880264 26302 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 551 FOLLOWER]: Advancing to term 552
I1014 19:27:18.882714 26302 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.883208 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 552 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.883347 26302 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 552 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.884354 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 552 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.884400 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 552 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.884562 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 552 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.884590 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 552 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.884976 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 552 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 552 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.885130 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 552 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 552 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:18.885183 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 552 election: Election decided. Result: candidate lost.
I1014 19:27:18.888028 26302 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:18.888108 26302 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Leader election lost for term 552. Reason: could not achieve majority
I1014 19:27:18.895777 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 551. Current term is 552. Ops: []
I1014 19:27:18.897204 26439 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 551.376, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I1014 19:27:18.897475 26439 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 LEADER]: Stepping down as leader of term 551
I1014 19:27:18.897539 26439 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:18.897737 26439 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 551.376, Last appended by leader: 376, Current term: 551, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.897958 26439 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:18.897996 26439 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:18.898105 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 551 FOLLOWER]: Advancing to term 552
I1014 19:27:18.979507 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:18.979583 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:18.979638 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 551 FOLLOWER]: Advancing to term 552
I1014 19:27:18.981751 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 552 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.982237 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 552 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:18.982388 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 552 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.983006 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 552 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.983070 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 552 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:18.983134 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:18.983178 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 552 FOLLOWER]: Advancing to term 553
I1014 19:27:18.983237 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 552: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.983994 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 552 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 552: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:18.985183 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.985653 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 553 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.985792 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 553 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:18.986024 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 552 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:18.986168 28640 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 552. Current term is 553.
I1014 19:27:18.986274 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 553 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:18.986390 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 552 FOLLOWER]: Advancing to term 553
I1014 19:27:18.986472 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:18.987614 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 553 candidate_status { last_received { term: 551 index: 376 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:18.987764 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 552 FOLLOWER]: Advancing to term 553
I1014 19:27:18.987818 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:18.988468 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 553 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 553.
I1014 19:27:18.989816 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 553.
I1014 19:27:18.990376 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 553 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:18.990435 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 553 election: Election decided. Result: candidate won.
I1014 19:27:18.990713 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:18.990767 26439 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Leader election won for term 553
I1014 19:27:18.990958 26439 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:18.991137 26439 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 551.376, Last appended by leader: 376, Current term: 553, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:18.991859 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 553 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
W1014 19:27:18.992360 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 552 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 552. Current term is 553.
I1014 19:27:18.992389 28786 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 552 election: Cancelling election due to peer responding with higher term
I1014 19:27:18.992588 26512 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:18.992833 26512 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:18.993070 26511 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:18.995052 26511 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:18.996415 26511 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:18.996868 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:18.996937 26390 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Leader election lost for term 552. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 552. Current term is 553.
I1014 19:27:18.998641 26511 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:19.001762 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 551 index: 376. Preceding OpId from leader: term: 553 index: 378. (index mismatch)
I1014 19:27:19.002497 26512 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I1014 19:27:19.004550 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.004992 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.008242 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 553 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 551 index: 376. Preceding OpId from leader: term: 553 index: 378. (index mismatch)
I1014 19:27:19.008538 25850 mvcc.cc:203] Tried to move safe_time back from 6306080518143463424 to 6306080518111158272. Current Snapshot: MvccSnapshot[committed={T|T < 6306080518143463424 or (T in {6306080518143463424})}]
I1014 19:27:19.008800 26512 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I1014 19:27:19.009264 26034 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.009465 26034 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.011291 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.011579 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.016950 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.017171 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.017300 26647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.017518 26647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.111340 26647 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 553 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:19.111408 26647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:27:19.111457 26647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 553 FOLLOWER]: Advancing to term 554
I1014 19:27:19.113657 26647 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.114133 26647 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 554 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.114287 26647 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 554 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:19.114800 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 554 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.114991 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 554 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.114969 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 554 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:19.115136 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 554 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.115321 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 554 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 554 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.117298 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 554 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 554 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.117362 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 554 election: Election decided. Result: candidate lost.
I1014 19:27:19.117621 26647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:27:19.117682 26647 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Leader election lost for term 554. Reason: could not achieve majority
I1014 19:27:19.132124 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 553 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.132378 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.132899 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 553. Current term is 554. Ops: []
I1014 19:27:19.133437 26034 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 553.378, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I1014 19:27:19.133658 26034 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 LEADER]: Stepping down as leader of term 553
I1014 19:27:19.133697 26034 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:19.133889 26034 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 553.378, Last appended by leader: 378, Current term: 553, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.134061 26034 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:19.134104 26034 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:19.134256 26034 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 553 FOLLOWER]: Advancing to term 554
I1014 19:27:19.213914 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:19.213990 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:19.214036 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 553 FOLLOWER]: Advancing to term 554
I1014 19:27:19.216228 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.216763 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 554 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.216920 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 554 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.217803 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 554 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.217972 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:19.219839 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 554 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:19.220039 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 554: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:19.220108 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 554 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 554.
I1014 19:27:19.220508 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 554 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 554: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:19.220767 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 554 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.220814 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 554 election: Election decided. Result: candidate won.
I1014 19:27:19.222007 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:19.222074 26390 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 FOLLOWER]: Leader election won for term 554
I1014 19:27:19.245782 26390 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:19.246146 26390 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 553.378, Last appended by leader: 378, Current term: 554, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.248342 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 554 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.248852 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.252959 26647 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:19.253041 26647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.207s (starting election)
I1014 19:27:19.253088 26647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 554 FOLLOWER]: Advancing to term 555
I1014 19:27:19.255328 26647 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.255846 26647 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 555 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.256018 26647 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 555 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:19.256639 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 555 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.256664 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 555 candidate_status { last_received { term: 553 index: 378 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:19.256788 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 554 FOLLOWER]: Advancing to term 555
I1014 19:27:19.256856 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 555 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.256845 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:19.257414 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 555 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 555 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.258702 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 555 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 555.
I1014 19:27:19.260674 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 555 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.260740 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 555 election: Election decided. Result: candidate won.
I1014 19:27:19.261014 26647 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:19.261077 26647 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 FOLLOWER]: Leader election won for term 555
I1014 19:27:19.261168 26647 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:19.261363 26647 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 553.378, Last appended by leader: 378, Current term: 555, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.262421 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.262615 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.265866 26722 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:19.267196 26724 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:19.269207 26724 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:19.273838 26724 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:19.274904 26724 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:19.276836 26722 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:19.277704 28825 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 LEADER]: Stepping down as leader of term 554
I1014 19:27:19.277756 28825 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:19.277796 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 555 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 553 index: 378. Preceding OpId from leader: term: 555 index: 380. (index mismatch)
I1014 19:27:19.277935 28825 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 554.379, Last appended by leader: 379, Current term: 554, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.278129 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:19.278167 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:19.278245 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 554 FOLLOWER]: Advancing to term 555
I1014 19:27:19.278417 26647 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I1014 19:27:19.280092 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.280273 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.280974 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 554 index: 379. Preceding OpId from leader: term: 555 index: 380. (index mismatch)
I1014 19:27:19.281760 26647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.281949 26647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.282613 26302 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 379, Last known committed idx: 378, Time since last communication: 0.000s
I1014 19:27:19.282732 26302 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 554.379 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 378
I1014 19:27:19.284668 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.284840 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.285637 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 378
I1014 19:27:19.285688 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 554.379
I1014 19:27:19.285728 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:19.286820 26722 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:19.290805 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.291013 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.292351 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.292541 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.294243 26722 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:19.306098 26722 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 554 vs 555: Illegal state: Replica cbc42c7d5db44e00883549e254bdd22e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.381330 26302 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 555 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.381647 26302 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.464496 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 555 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:19.464576 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:19.464625 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 555 FOLLOWER]: Advancing to term 556
I1014 19:27:19.466691 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.467159 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 556 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:19.467296 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 556 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.467883 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 556 candidate_status { last_received { term: 555 index: 380 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:19.468065 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 556 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.468427 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 556 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 556 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.468899 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 556 candidate_status { last_received { term: 555 index: 380 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:19.469096 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 556 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.469585 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 556 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 556 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.469645 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 556 election: Election decided. Result: candidate lost.
I1014 19:27:19.471483 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:19.471580 26439 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Leader election lost for term 556. Reason: could not achieve majority
I1014 19:27:19.475375 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 555. Current term is 556. Ops: []
I1014 19:27:19.476310 26647 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 555.380, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I1014 19:27:19.476599 26647 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 LEADER]: Stepping down as leader of term 555
I1014 19:27:19.476639 26647 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:19.476799 26647 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 555.380, Last appended by leader: 380, Current term: 555, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.476965 26647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:19.477001 26647 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:19.477108 26647 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 555 FOLLOWER]: Advancing to term 556
I1014 19:27:19.559193 26390 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:19.559285 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:19.559366 26390 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 555 FOLLOWER]: Advancing to term 556
I1014 19:27:19.561720 26390 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.562247 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 556 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.562407 26390 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 556 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.562999 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 556 candidate_status { last_received { term: 555 index: 380 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.563168 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 556: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:19.563417 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 556 candidate_status { last_received { term: 555 index: 380 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:19.563592 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:19.565742 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 556 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 556.
I1014 19:27:19.566442 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 556 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 556: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:19.566993 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 556 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.567050 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 556 election: Election decided. Result: candidate won.
I1014 19:27:19.567246 26390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (election complete)
I1014 19:27:19.567348 26390 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 FOLLOWER]: Leader election won for term 556
I1014 19:27:19.567477 26390 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:19.567737 26390 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 555.380, Last appended by leader: 380, Current term: 556, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.569095 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.569346 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.569581 26722 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:19.571465 26722 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:19.572784 26722 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:19.573633 26722 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:19.576095 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 555 index: 380. Preceding OpId from leader: term: 556 index: 382. (index mismatch)
I1014 19:27:19.576644 26720 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I1014 19:27:19.578147 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.578407 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.581082 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 556 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 555 index: 380. Preceding OpId from leader: term: 556 index: 382. (index mismatch)
I1014 19:27:19.581609 26390 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I1014 19:27:19.582021 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.582228 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.583704 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.583873 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.584240 26647 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.584467 26647 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.584719 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.584956 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.613822 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:19.683370 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 556 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:19.683463 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:19.683550 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 556 FOLLOWER]: Advancing to term 557
I1014 19:27:19.684033 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:19.684104 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:19.684156 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 556 FOLLOWER]: Advancing to term 557
I1014 19:27:19.685571 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.685976 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 557 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:19.685849 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.686120 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 557 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.686344 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 557 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.686491 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 557 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:19.686704 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 557 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:19.686874 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 557: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:19.686964 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 557 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:19.687147 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 557 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.687228 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 557 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 557: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:19.687238 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 557 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:19.687438 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 557 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.687682 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 557 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.687855 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 557: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:19.688022 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 557 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 557 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.687736 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 557 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 557 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:19.688172 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 557 election: Election decided. Result: candidate lost.
I1014 19:27:19.688385 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:19.688438 26439 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Leader election lost for term 557. Reason: could not achieve majority
I1014 19:27:19.688670 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 557 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 557: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:19.693081 26814 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 556 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.693384 26814 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.693691 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 556. Current term is 557. Ops: []
I1014 19:27:19.694211 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 556. Current term is 557. Ops: []
I1014 19:27:19.694185 26720 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 556.382, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I1014 19:27:19.694406 26720 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Stepping down as leader of term 556
I1014 19:27:19.694447 26720 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:19.694665 26720 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 556.382, Last appended by leader: 382, Current term: 556, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.694929 26720 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:19.694962 26390 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 556.382, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I1014 19:27:19.695123 26720 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:19.695281 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 556 FOLLOWER]: Advancing to term 557
I1014 19:27:19.698751 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 557 election: Election decided. Result: candidate lost.
I1014 19:27:19.699100 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:19.699164 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Leader election lost for term 557. Reason: could not achieve majority
I1014 19:27:19.787722 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 557 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:19.787796 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:19.787844 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 557 FOLLOWER]: Advancing to term 558
I1014 19:27:19.789862 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.790328 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 558 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.790496 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 558 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.791064 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 558 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:19.791177 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 558 candidate_status { last_received { term: 556 index: 382 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:19.791263 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 557 FOLLOWER]: Advancing to term 558
I1014 19:27:19.791314 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 557 FOLLOWER]: Advancing to term 558
I1014 19:27:19.791333 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.186s (vote granted)
I1014 19:27:19.791375 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.195s (vote granted)
I1014 19:27:19.793473 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 558 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 558.
I1014 19:27:19.793545 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 558.
I1014 19:27:19.794044 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 558 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:19.794109 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 558 election: Election decided. Result: candidate won.
I1014 19:27:19.794304 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:19.794358 26720 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 FOLLOWER]: Leader election won for term 558
I1014 19:27:19.794576 26720 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:19.794598 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 558 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:19.794800 26720 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 556.382, Last appended by leader: 382, Current term: 558, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:19.795727 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.795958 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.796152 26722 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:19.798329 26722 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:19.800343 26722 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:19.807639 26722 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:19.810021 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 558 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 556 index: 382. Preceding OpId from leader: term: 558 index: 384. (index mismatch)
I1014 19:27:19.810528 26390 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I1014 19:27:19.811547 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 556 index: 382. Preceding OpId from leader: term: 558 index: 384. (index mismatch)
I1014 19:27:19.812111 26390 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I1014 19:27:19.812225 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.812501 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.813791 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.814661 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.814867 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.815795 26390 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.815977 26390 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:19.817790 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.818027 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.820297 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 558 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:19.820508 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:19.825994 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.009699 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 558 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:20.009785 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:20.009845 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 558 FOLLOWER]: Advancing to term 559
I1014 19:27:20.012054 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.012622 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 559 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.012830 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 559 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.013700 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 559 candidate_status { last_received { term: 558 index: 384 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.013881 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 559 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.014096 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 559 candidate_status { last_received { term: 558 index: 384 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.014195 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 559 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 559 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.014273 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 559 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.014628 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 559 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 559 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.014693 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 559 election: Election decided. Result: candidate lost.
I1014 19:27:20.014883 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:20.014948 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Leader election lost for term 559. Reason: could not achieve majority
I1014 19:27:20.020591 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 558. Current term is 559. Ops: []
I1014 19:27:20.021275 26814 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 558.384, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I1014 19:27:20.021548 26847 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 LEADER]: Stepping down as leader of term 558
I1014 19:27:20.021600 26847 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:20.021781 26847 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 558.384, Last appended by leader: 384, Current term: 558, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.021965 26847 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:20.022003 26847 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:20.022128 26847 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 558 FOLLOWER]: Advancing to term 559
I1014 19:27:20.104492 26439 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:20.104578 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:20.104629 26439 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 558 FOLLOWER]: Advancing to term 559
I1014 19:27:20.106428 26439 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.106947 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 559 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.107103 26439 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 559 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:20.107635 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 559 candidate_status { last_received { term: 558 index: 384 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.107791 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (vote granted)
I1014 19:27:20.107736 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 559 candidate_status { last_received { term: 558 index: 384 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:20.107928 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 559: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.108458 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 559 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 559: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.109887 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 559.
I1014 19:27:20.110556 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 559 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.110610 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 559 election: Election decided. Result: candidate won.
I1014 19:27:20.110934 26439 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:20.110985 26439 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 FOLLOWER]: Leader election won for term 559
I1014 19:27:20.111270 26439 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:20.111526 26439 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 558.384, Last appended by leader: 384, Current term: 559, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.112589 26894 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.112920 26894 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.113517 26895 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:20.115495 26895 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:20.116968 26895 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:20.117947 26895 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:20.120496 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 558 index: 384. Preceding OpId from leader: term: 559 index: 386. (index mismatch)
I1014 19:27:20.120527 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 558 index: 384. Preceding OpId from leader: term: 559 index: 386. (index mismatch)
I1014 19:27:20.121098 26894 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I1014 19:27:20.121309 26439 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I1014 19:27:20.122972 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.123186 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.123131 26847 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.123314 26847 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.126145 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.126334 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.127897 26847 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.128074 26847 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.127063 26894 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.128376 26894 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.133954 26894 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 559 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.134281 26894 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.228360 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:20.228438 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:20.228523 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 559 FOLLOWER]: Advancing to term 560
I1014 19:27:20.230011 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.230535 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 560 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.230693 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 560 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.231237 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 560 candidate_status { last_received { term: 559 index: 386 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.231423 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 560 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.231753 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 560 candidate_status { last_received { term: 559 index: 386 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.231937 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 560 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.232293 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 560 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 560 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.232439 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 560 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 560 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.232491 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 560 election: Election decided. Result: candidate lost.
I1014 19:27:20.232655 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:20.232707 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Leader election lost for term 560. Reason: could not achieve majority
I1014 19:27:20.248973 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 559. Current term is 560. Ops: []
I1014 19:27:20.249476 26897 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 559.386, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I1014 19:27:20.249794 26897 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 LEADER]: Stepping down as leader of term 559
I1014 19:27:20.249843 26897 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:20.250025 26897 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 559.386, Last appended by leader: 386, Current term: 559, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.250254 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:20.250294 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:20.250413 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 559 FOLLOWER]: Advancing to term 560
I1014 19:27:20.304056 26847 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:20.304141 26847 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:20.304196 26847 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 559 FOLLOWER]: Advancing to term 560
I1014 19:27:20.306339 26847 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.306890 26847 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 560 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.307058 26847 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 560 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:20.307600 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 560 candidate_status { last_received { term: 559 index: 386 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.307682 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 560 candidate_status { last_received { term: 559 index: 386 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:20.307747 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (vote granted)
I1014 19:27:20.307833 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 560: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.308238 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 560 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 560: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.309146 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 560.
I1014 19:27:20.310029 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 560 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.310079 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 560 election: Election decided. Result: candidate won.
I1014 19:27:20.310279 26847 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:20.310380 26847 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 FOLLOWER]: Leader election won for term 560
I1014 19:27:20.310534 26847 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:20.310765 26847 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 559.386, Last appended by leader: 386, Current term: 560, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.312175 26814 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.312351 26814 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.312551 26722 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:20.313851 26722 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:20.315100 26722 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:20.315951 26722 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:20.323451 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 559 index: 386. Preceding OpId from leader: term: 560 index: 388. (index mismatch)
I1014 19:27:20.323999 26847 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I1014 19:27:20.324563 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 559 index: 386. Preceding OpId from leader: term: 560 index: 388. (index mismatch)
I1014 19:27:20.325042 26847 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I1014 19:27:20.325496 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.325726 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.326611 26814 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.326791 26814 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.327060 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.327309 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.328166 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.328337 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.328526 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.328722 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.329041 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 560 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.329309 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.428009 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:20.428088 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:20.428145 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 560 FOLLOWER]: Advancing to term 561
I1014 19:27:20.430310 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.430878 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 561 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.431543 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 561 candidate_status { last_received { term: 560 index: 388 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.431759 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 561 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.432176 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 561 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 561 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.432466 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 561 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.433071 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 561 candidate_status { last_received { term: 560 index: 388 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.433226 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 561 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.433548 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 561 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 561 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.433609 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 561 election: Election decided. Result: candidate lost.
I1014 19:27:20.433776 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:20.433869 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Leader election lost for term 561. Reason: could not achieve majority
I1014 19:27:20.449975 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 560. Current term is 561. Ops: []
I1014 19:27:20.450812 26814 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 560.388, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I1014 19:27:20.451062 26814 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 LEADER]: Stepping down as leader of term 560
I1014 19:27:20.451103 26814 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:20.451308 26814 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 560.388, Last appended by leader: 388, Current term: 560, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.451540 26814 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:20.451583 26814 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:20.451712 26814 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 560 FOLLOWER]: Advancing to term 561
I1014 19:27:20.517354 26897 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:20.517441 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:20.517500 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 560 FOLLOWER]: Advancing to term 561
I1014 19:27:20.519682 26897 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.520212 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 561 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.520406 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 561 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:20.520974 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 561 candidate_status { last_received { term: 560 index: 388 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.521140 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.132s (vote granted)
I1014 19:27:20.521178 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 561 candidate_status { last_received { term: 560 index: 388 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:20.521409 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 561: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.521870 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 561 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 561: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:20.523139 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 561 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 561.
I1014 19:27:20.523519 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 561 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.523576 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 561 election: Election decided. Result: candidate won.
I1014 19:27:20.523767 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:20.523826 26897 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 FOLLOWER]: Leader election won for term 561
I1014 19:27:20.523941 26897 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:20.524108 26897 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 560.388, Last appended by leader: 388, Current term: 561, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.525051 26894 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.525234 26894 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.525445 26895 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:20.527153 26895 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:20.528376 26895 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:20.529161 26895 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:20.531422 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 561 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 560 index: 388. Preceding OpId from leader: term: 561 index: 390. (index mismatch)
I1014 19:27:20.531616 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 560 index: 388. Preceding OpId from leader: term: 561 index: 390. (index mismatch)
I1014 19:27:20.532058 26897 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I1014 19:27:20.532399 26894 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I1014 19:27:20.534193 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.534409 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.535554 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.535759 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.539098 25848 mvcc.cc:203] Tried to move safe_time back from 6306080524411318272 to 6306080524389347328. Current Snapshot: MvccSnapshot[committed={T|T < 6306080524411318272 or (T in {6306080524411318272})}]
I1014 19:27:20.539311 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.539495 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.540081 26439 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.540148 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.540343 26439 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.540369 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.541251 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 561 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.541450 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.609977 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
I1014 19:27:20.799034 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 561 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:20.799111 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:20.799158 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 561 FOLLOWER]: Advancing to term 562
I1014 19:27:20.801323 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.801854 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 562 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.802000 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 562 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:20.803588 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 561. Current term is 562. Ops: []
I1014 19:27:20.803534 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 562 candidate_status { last_received { term: 561 index: 390 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.803711 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 562 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.804083 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 562 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 562 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.804244 26897 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 561.390, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I1014 19:27:20.804491 26896 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 LEADER]: Stepping down as leader of term 561
I1014 19:27:20.804539 26896 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:20.804839 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 562 candidate_status { last_received { term: 561 index: 390 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:20.804826 26896 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 561.390, Last appended by leader: 390, Current term: 561, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.805007 26896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:20.805022 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 562 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.805048 26896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:20.805187 26896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 561 FOLLOWER]: Advancing to term 562
I1014 19:27:20.807399 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 562 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 562 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.807497 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 562 election: Election decided. Result: candidate lost.
I1014 19:27:20.807729 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:20.807792 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Leader election lost for term 562. Reason: could not achieve majority
I1014 19:27:20.826719 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:20.826797 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:20.826856 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 561 FOLLOWER]: Advancing to term 562
I1014 19:27:20.829020 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.829557 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 562 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.829726 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 562 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.830312 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 562 candidate_status { last_received { term: 561 index: 390 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:20.830312 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 562 candidate_status { last_received { term: 561 index: 390 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.830498 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 562: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:20.830495 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:20.830991 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 562 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 562: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:20.832532 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 562 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 562.
I1014 19:27:20.833040 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 562 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:20.833096 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 562 election: Election decided. Result: candidate won.
I1014 19:27:20.833482 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:20.833586 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 FOLLOWER]: Leader election won for term 562
I1014 19:27:20.833945 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:20.834198 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 561.390, Last appended by leader: 390, Current term: 562, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.835659 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.835913 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.838627 27028 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:20.840029 27028 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:20.842546 27028 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:20.843204 27028 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:20.845111 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 561 index: 390. Preceding OpId from leader: term: 562 index: 392. (index mismatch)
I1014 19:27:20.845211 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 562 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 561 index: 390. Preceding OpId from leader: term: 562 index: 392. (index mismatch)
I1014 19:27:20.845598 27027 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I1014 19:27:20.845783 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I1014 19:27:20.847159 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.847342 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.847368 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.847580 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.861238 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.861506 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.861735 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.861981 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:20.861896 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.862080 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.863806 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 562 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:20.864012 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:20.964361 26896 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 562 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:20.964437 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:20.964511 26896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 562 FOLLOWER]: Advancing to term 563
I1014 19:27:20.966575 26896 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.967046 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 563 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:20.967180 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 563 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:20.967767 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 563 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:20.967963 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 563 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.968425 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 563 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:20.968628 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 563 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.969009 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 563 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 563 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.969172 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 563 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 563 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:20.969233 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 563 election: Election decided. Result: candidate lost.
I1014 19:27:20.970491 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:27:20.970551 26896 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Leader election lost for term 563. Reason: could not achieve majority
I1014 19:27:20.979364 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 562. Current term is 563. Ops: []
I1014 19:27:20.979851 27027 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 562.392, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I1014 19:27:20.980067 26735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 LEADER]: Stepping down as leader of term 562
I1014 19:27:20.980125 26735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:20.980355 26735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 562.392, Last appended by leader: 392, Current term: 562, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:20.980585 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:20.980634 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:20.980792 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 562 FOLLOWER]: Advancing to term 563
I1014 19:27:21.054661 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:21.054742 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:21.054816 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 562 FOLLOWER]: Advancing to term 563
I1014 19:27:21.056917 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 563 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.057389 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 563 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.057559 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 563 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.058151 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 563 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:21.058225 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:21.058281 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 563 FOLLOWER]: Advancing to term 564
I1014 19:27:21.058377 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 563 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.060360 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.060876 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 564 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.061029 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 564 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.061086 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 563. Current term is 564.
I1014 19:27:21.061298 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 563 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.061511 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 563: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:21.061828 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 564 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.061952 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Advancing to term 564
I1014 19:27:21.062021 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:21.062199 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 563 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 563 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 563: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
W1014 19:27:21.062332 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 563 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 563. Current term is 564.
I1014 19:27:21.062367 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 563 election: Cancelling election due to peer responding with higher term
I1014 19:27:21.062521 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 564 candidate_status { last_received { term: 562 index: 392 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:21.062641 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 563 FOLLOWER]: Advancing to term 564
I1014 19:27:21.062691 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:21.064035 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 564.
I1014 19:27:21.064651 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 564 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 564.
I1014 19:27:21.064752 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 564 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.064816 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 564 election: Election decided. Result: candidate won.
I1014 19:27:21.065021 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:21.065050 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 564 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.065064 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (election complete)
I1014 19:27:21.065084 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Leader election won for term 564
I1014 19:27:21.065119 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 564 FOLLOWER]: Leader election lost for term 563. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 563. Current term is 564.
I1014 19:27:21.065187 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:21.065389 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 562.392, Last appended by leader: 392, Current term: 564, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.066599 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.066799 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.066977 27028 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:21.068680 27028 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:21.069918 27028 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:21.070720 27028 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:21.072988 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 564 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 562 index: 392. Preceding OpId from leader: term: 564 index: 394. (index mismatch)
I1014 19:27:21.073143 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 562 index: 392. Preceding OpId from leader: term: 564 index: 394. (index mismatch)
I1014 19:27:21.073953 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I1014 19:27:21.074240 27027 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I1014 19:27:21.076001 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.076210 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.077801 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.077989 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.080178 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.080343 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.080307 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.080519 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.088569 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.088811 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.089493 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 564 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.089691 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.189795 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 564 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:21.189869 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:21.189913 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 564 FOLLOWER]: Advancing to term 565
I1014 19:27:21.192049 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.192530 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 565 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.192667 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 565 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.193365 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 565 candidate_status { last_received { term: 564 index: 394 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.193373 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 565 candidate_status { last_received { term: 564 index: 394 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.193573 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 565 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.193584 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 565 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.194175 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 565 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 565 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.194258 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 565 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 565 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.194340 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 565 election: Election decided. Result: candidate lost.
I1014 19:27:21.194553 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:21.194641 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Leader election lost for term 565. Reason: could not achieve majority
I1014 19:27:21.204226 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 564. Current term is 565. Ops: []
I1014 19:27:21.204778 27027 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 564.394, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I1014 19:27:21.205032 27029 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 LEADER]: Stepping down as leader of term 564
I1014 19:27:21.205090 27029 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:21.205354 27029 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 564.394, Last appended by leader: 394, Current term: 564, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.205596 27029 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:21.205648 27029 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:21.205801 27029 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 564 FOLLOWER]: Advancing to term 565
I1014 19:27:21.271001 26896 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:21.271086 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (starting election)
I1014 19:27:21.271152 26896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 564 FOLLOWER]: Advancing to term 565
I1014 19:27:21.273332 26896 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.273878 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 565 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.274046 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 565 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.274605 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 565 candidate_status { last_received { term: 564 index: 394 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:21.274791 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 565: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.275156 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 565 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 565: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.275089 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 565 candidate_status { last_received { term: 564 index: 394 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.275261 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:21.277304 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 565.
I1014 19:27:21.277704 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 565 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.277760 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 565 election: Election decided. Result: candidate won.
I1014 19:27:21.278132 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:21.278234 26896 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 FOLLOWER]: Leader election won for term 565
I1014 19:27:21.278445 26896 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:21.278718 26896 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 564.394, Last appended by leader: 394, Current term: 565, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.280369 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.280575 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.282209 27092 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:21.284341 27092 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:21.285794 27092 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:21.286675 27092 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:21.289103 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 564 index: 394. Preceding OpId from leader: term: 565 index: 396. (index mismatch)
I1014 19:27:21.289692 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 564 index: 394. Preceding OpId from leader: term: 565 index: 396. (index mismatch)
I1014 19:27:21.289971 26897 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I1014 19:27:21.290171 26896 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I1014 19:27:21.291745 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.291941 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.291860 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.292065 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.295073 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.295255 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.296072 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.296268 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.296710 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.296871 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.298154 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 565 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.298332 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.476310 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:21.476393 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:21.476478 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 565 FOLLOWER]: Advancing to term 566
I1014 19:27:21.478631 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.479123 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 566 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.479280 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 566 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.480149 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 566 candidate_status { last_received { term: 565 index: 396 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.480203 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 566 candidate_status { last_received { term: 565 index: 396 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.480340 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 566 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.480383 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 566 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.480799 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 566 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 566 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.480957 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 566 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 566 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.481007 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 566 election: Election decided. Result: candidate lost.
I1014 19:27:21.481173 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:21.481225 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Leader election lost for term 566. Reason: could not achieve majority
I1014 19:27:21.490933 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 565. Current term is 566. Ops: []
I1014 19:27:21.491472 26897 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 565.396, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I1014 19:27:21.491714 26897 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 LEADER]: Stepping down as leader of term 565
I1014 19:27:21.491752 26897 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:21.491930 26897 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 565.396, Last appended by leader: 396, Current term: 565, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.492112 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:21.492144 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:21.492251 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 565 FOLLOWER]: Advancing to term 566
I1014 19:27:21.492652 27029 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:21.492724 27029 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:21.492774 27029 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 565 FOLLOWER]: Advancing to term 566
I1014 19:27:21.494751 27029 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.495257 27029 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 566 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.495458 27029 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 566 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.496129 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 566 candidate_status { last_received { term: 565 index: 396 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.496130 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 566 candidate_status { last_received { term: 565 index: 396 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:21.496284 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:27:21.496366 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 566: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.496870 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 566 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 566: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.498257 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 566.
I1014 19:27:21.498565 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 566 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.498612 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 566 election: Election decided. Result: candidate won.
I1014 19:27:21.498792 27029 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:21.498844 27029 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 FOLLOWER]: Leader election won for term 566
I1014 19:27:21.498916 27029 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:21.499088 27029 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 396, Committed index: 396, Last appended: 565.396, Last appended by leader: 396, Current term: 566, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.500257 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.500497 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.500677 27028 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:21.502751 27028 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:21.504182 27028 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:21.506084 27028 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:21.508602 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 565 index: 396. Preceding OpId from leader: term: 566 index: 398. (index mismatch)
I1014 19:27:21.508680 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 565 index: 396. Preceding OpId from leader: term: 566 index: 398. (index mismatch)
I1014 19:27:21.509183 27029 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I1014 19:27:21.509326 27027 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 397, Last known committed idx: 396, Time since last communication: 0.000s
I1014 19:27:21.511188 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.511397 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.511600 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.511803 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.513170 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.513362 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.513659 26896 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.513842 26896 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.514665 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.514833 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.515717 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 566 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.515890 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.616171 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:21.616261 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:21.616315 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 566 FOLLOWER]: Advancing to term 567
I1014 19:27:21.618227 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.618615 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 567 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.618731 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 567 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.619220 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 567 candidate_status { last_received { term: 566 index: 398 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.619298 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 567 candidate_status { last_received { term: 566 index: 398 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.619427 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 567 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.619505 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 567 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.619910 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:21.619899 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 567 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 567 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.620002 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 567 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 567 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:21.620062 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 567 election: Election decided. Result: candidate lost.
I1014 19:27:21.620429 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:21.620512 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Leader election lost for term 567. Reason: could not achieve majority
I1014 19:27:21.632254 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 566. Current term is 567. Ops: []
I1014 19:27:21.632784 27029 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 566.398, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I1014 19:27:21.633047 27029 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 LEADER]: Stepping down as leader of term 566
I1014 19:27:21.633088 27029 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:21.633301 27029 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 566.398, Last appended by leader: 398, Current term: 566, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.633508 27029 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:21.633553 27029 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:21.633687 27029 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 566 FOLLOWER]: Advancing to term 567
I1014 19:27:21.702088 26896 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:21.702189 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:21.702253 26896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 566 FOLLOWER]: Advancing to term 567
I1014 19:27:21.704581 26896 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.705212 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 567 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.705379 26896 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 567 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.705991 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 567 candidate_status { last_received { term: 566 index: 398 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:21.706192 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 567: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.706511 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 567 candidate_status { last_received { term: 566 index: 398 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.706665 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (vote granted)
I1014 19:27:21.706948 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 567 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 567: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.708698 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 567 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 567.
I1014 19:27:21.709182 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 567 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.709234 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 567 election: Election decided. Result: candidate won.
I1014 19:27:21.709436 26896 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:21.709493 26896 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 FOLLOWER]: Leader election won for term 567
I1014 19:27:21.709597 26896 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:21.709789 26896 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 566.398, Last appended by leader: 398, Current term: 567, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.711215 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.711402 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.711632 27092 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:21.713340 27092 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:21.714716 27092 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:21.716070 27092 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:21.718317 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 566 index: 398. Preceding OpId from leader: term: 567 index: 400. (index mismatch)
I1014 19:27:21.718376 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 567 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 566 index: 398. Preceding OpId from leader: term: 567 index: 400. (index mismatch)
I1014 19:27:21.718796 26896 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I1014 19:27:21.719139 26896 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I1014 19:27:21.720450 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.720665 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.721546 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.721743 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.721791 27098 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.722148 27098 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.722789 27098 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.722820 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.722990 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.722990 27098 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.724483 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 567 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.724678 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.822867 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:21.822950 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (starting election)
I1014 19:27:21.823012 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 567 FOLLOWER]: Advancing to term 568
I1014 19:27:21.825098 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.825402 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 567. Current term is 568. Ops: []
I1014 19:27:21.825618 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 568 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.825750 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 568 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:21.826063 26896 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 567.400, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I1014 19:27:21.826314 26896 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 LEADER]: Stepping down as leader of term 567
I1014 19:27:21.826349 26896 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:21.826539 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 567 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:21.826607 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:21.826653 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 567 FOLLOWER]: Advancing to term 568
I1014 19:27:21.826563 26896 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 567.400, Last appended by leader: 400, Current term: 567, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.826759 26896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:21.826798 26896 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:21.826916 26896 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 567 FOLLOWER]: Advancing to term 568
I1014 19:27:21.828752 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.829022 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 568 candidate_status { last_received { term: 567 index: 400 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:21.829183 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 568: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:21.829202 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 568 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.829332 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 568 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:21.829582 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 568 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 568: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:21.830070 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 568 candidate_status { last_received { term: 567 index: 400 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.830211 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:21.830291 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 568 candidate_status { last_received { term: 567 index: 400 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:21.830454 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 568: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.830512 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 568 candidate_status { last_received { term: 567 index: 400 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:21.830813 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 568 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 568: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:21.832212 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 568.
I1014 19:27:21.832309 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 568 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:21.832746 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 568 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:21.832808 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 568 election: Election decided. Result: candidate won.
I1014 19:27:21.832821 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 568 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 568 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:21.832901 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 568 election: Election decided. Result: candidate lost.
I1014 19:27:21.833114 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:21.833184 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 FOLLOWER]: Leader election won for term 568
I1014 19:27:21.833225 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:21.833287 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Leader election lost for term 568. Reason: could not achieve majority
I1014 19:27:21.833278 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:21.833477 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 567.400, Last appended by leader: 400, Current term: 568, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:21.835021 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.835201 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.835383 27028 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:21.837391 27028 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:21.838860 27028 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:21.839826 27028 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:21.842783 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 567 index: 400. Preceding OpId from leader: term: 568 index: 402. (index mismatch)
I1014 19:27:21.842783 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 567 index: 400. Preceding OpId from leader: term: 568 index: 402. (index mismatch)
I1014 19:27:21.843647 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I1014 19:27:21.844481 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I1014 19:27:21.845708 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.845898 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.847440 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.847729 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.848799 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.849045 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.850358 27029 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.850554 27029 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:21.851642 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.851831 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:21.853209 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 568 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:21.853387 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.027269 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:22.027340 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:22.027390 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 568 FOLLOWER]: Advancing to term 569
I1014 19:27:22.029400 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 569 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.029870 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 569 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.030045 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 569 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.030992 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 569 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.031185 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.031111 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 569 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.031302 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.031584 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 569 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.031718 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 569 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.031778 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 569 election: Election decided. Result: candidate lost.
I1014 19:27:22.032091 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:22.032146 26720 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 569 FOLLOWER]: Leader election lost for term 569. Reason: could not achieve majority
I1014 19:27:22.038050 26897 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:22.038112 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:22.038152 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 568 FOLLOWER]: Advancing to term 569
I1014 19:27:22.040127 26897 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 569 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.040591 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 569 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.040760 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 569 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.041388 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 569 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.041577 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 569 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 569: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.042050 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 569 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.042232 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.042390 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 569 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 568. Current term is 569. Ops: []
I1014 19:27:22.042495 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 569 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 569 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 569: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.042906 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 569 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 569 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.042958 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 569 election: Election decided. Result: candidate lost.
I1014 19:27:22.042873 26735 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 568.402, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I1014 19:27:22.043084 26735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Stepping down as leader of term 568
I1014 19:27:22.043118 26735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:22.043341 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:22.043390 26897 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 569 FOLLOWER]: Leader election lost for term 569. Reason: could not achieve majority
I1014 19:27:22.043267 26735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 568.402, Last appended by leader: 402, Current term: 568, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.043463 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:22.043500 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:22.043602 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 568 FOLLOWER]: Advancing to term 569
I1014 19:27:22.129431 27027 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 569 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:22.129536 27027 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:22.129588 27027 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 569 FOLLOWER]: Advancing to term 570
I1014 19:27:22.131688 27027 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.132196 27027 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 570 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.132364 27027 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 570 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.132990 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 570 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.133146 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 569 FOLLOWER]: Advancing to term 570
I1014 19:27:22.133215 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.150s (vote granted)
I1014 19:27:22.134208 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 570 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.134352 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 569 FOLLOWER]: Advancing to term 570
I1014 19:27:22.134413 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.212s (vote granted)
I1014 19:27:22.135260 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 570 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 570.
I1014 19:27:22.135762 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 570 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.135826 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 570 election: Election decided. Result: candidate won.
I1014 19:27:22.136014 27027 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:22.136080 27027 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 FOLLOWER]: Leader election won for term 570
I1014 19:27:22.136165 27027 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:22.136250 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 570 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 570.
I1014 19:27:22.136319 26720 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 570 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:22.136368 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.202s (starting election)
I1014 19:27:22.136410 26720 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 570 FOLLOWER]: Advancing to term 571
I1014 19:27:22.136359 27027 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 568.402, Last appended by leader: 402, Current term: 570, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.136585 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 570 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.137347 27027 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 570 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.137547 27027 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.137770 27028 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:22.137749 26720 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.138187 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 571 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.138329 26720 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 571 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.138780 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 571 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.138919 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 570 FOLLOWER]: Advancing to term 571
I1014 19:27:22.138985 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (vote granted)
I1014 19:27:22.139875 27028 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:22.139937 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 571 candidate_status { last_received { term: 568 index: 402 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.140095 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 571 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.140466 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 571 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 571 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.140820 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 571.
I1014 19:27:22.141084 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 571 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.141135 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 571 election: Election decided. Result: candidate won.
I1014 19:27:22.141270 27028 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:22.141301 26720 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.166s (election complete)
I1014 19:27:22.141355 26720 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 FOLLOWER]: Leader election won for term 571
I1014 19:27:22.141592 26720 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:22.141777 26720 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 568.402, Last appended by leader: 402, Current term: 571, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.142874 27028 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:22.142612 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.143023 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.143532 27338 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:22.145028 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 570. Current term is 571. Ops: []
I1014 19:27:22.145527 27027 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I1014 19:27:22.145737 26735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 LEADER]: Stepping down as leader of term 570
I1014 19:27:22.145779 26735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:22.145936 26735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 402, Committed index: 402, Last appended: 570.404, Last appended by leader: 404, Current term: 570, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.146107 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:22.146142 27338 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:22.146139 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:22.146217 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 570 FOLLOWER]: Advancing to term 571
I1014 19:27:22.146299 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 570. Current term is 571. Ops: []
I1014 19:27:22.147327 27338 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:22.148073 27338 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:22.149857 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 571 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 570 index: 404. Preceding OpId from leader: term: 571 index: 404. (term mismatch)
I1014 19:27:22.149902 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 403
I1014 19:27:22.149924 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted WRITE_OP operation due to leader change: 570.404
I1014 19:27:22.149989 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 568 index: 402. Preceding OpId from leader: term: 571 index: 404. (index mismatch)
I1014 19:27:22.150262 27028 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 570 vs 571: Aborted: Transaction aborted by new leader
I1014 19:27:22.150472 27337 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I1014 19:27:22.150892 26720 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 403, Last known committed idx: 402, Time since last communication: 0.000s
I1014 19:27:22.151037 26720 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 570.403 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 402
I1014 19:27:22.151891 29011 pending_rounds.cc:76] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting all transactions after (but not including) 402
I1014 19:27:22.151919 29011 pending_rounds.cc:97] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Aborting uncommitted NO_OP operation due to leader change: 570.403
I1014 19:27:22.151957 29011 raft_consensus.cc:2632] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 571 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:22.152695 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.152932 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.152873 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.153131 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.162617 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.162829 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.163327 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.163555 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.163039 26720 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.163779 26720 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.165083 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 571 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.165297 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.265580 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 571 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:22.265667 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:22.265723 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 571 FOLLOWER]: Advancing to term 572
I1014 19:27:22.267184 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.267586 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 572 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.267731 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 572 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.268435 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 572 candidate_status { last_received { term: 571 index: 404 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.268435 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 572 candidate_status { last_received { term: 571 index: 404 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.268649 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 572 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.268664 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 572 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.269011 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 572 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 572 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.269148 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 572 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 572 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.269196 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 572 election: Election decided. Result: candidate lost.
I1014 19:27:22.269358 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:22.269412 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Leader election lost for term 572. Reason: could not achieve majority
I1014 19:27:22.282013 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 571. Current term is 572. Ops: []
I1014 19:27:22.282596 27339 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 571.404, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I1014 19:27:22.282923 27337 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Stepping down as leader of term 571
I1014 19:27:22.282968 27337 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:22.283149 27337 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 571.404, Last appended by leader: 404, Current term: 571, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.283336 27337 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:22.283376 27337 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:22.283529 27337 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 571 FOLLOWER]: Advancing to term 572
I1014 19:27:22.346546 26897 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:22.346635 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:22.346690 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 571 FOLLOWER]: Advancing to term 572
I1014 19:27:22.348716 26897 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.349223 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 572 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.349380 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 572 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.350790 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 572 candidate_status { last_received { term: 571 index: 404 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.350967 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 572: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:22.351691 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 572 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 572: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:22.351894 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 572 candidate_status { last_received { term: 571 index: 404 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.352049 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (vote granted)
I1014 19:27:22.353492 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 572 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 572.
I1014 19:27:22.353786 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 572 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.353838 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 572 election: Election decided. Result: candidate won.
I1014 19:27:22.354019 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:22.354077 26897 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 FOLLOWER]: Leader election won for term 572
I1014 19:27:22.354332 26897 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:22.354559 26897 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 571.404, Last appended by leader: 404, Current term: 572, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.355460 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.355660 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.356269 27446 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:22.358292 27446 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:22.359707 27446 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:22.360558 27446 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:22.362912 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 571 index: 404. Preceding OpId from leader: term: 572 index: 406. (index mismatch)
I1014 19:27:22.362920 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 572 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 571 index: 404. Preceding OpId from leader: term: 572 index: 406. (index mismatch)
I1014 19:27:22.363399 26897 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I1014 19:27:22.363646 27444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I1014 19:27:22.365167 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.365489 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.365840 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.366119 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.367233 26897 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.367403 26897 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.368072 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.368320 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.369030 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.369038 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 572 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.369200 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.369220 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.468156 27337 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 572 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:22.468227 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:22.468277 27337 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 572 FOLLOWER]: Advancing to term 573
I1014 19:27:22.470041 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:22.470106 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:22.470150 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 572 FOLLOWER]: Advancing to term 573
I1014 19:27:22.470336 27337 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.470824 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 573 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.470965 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 573 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.472018 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 573 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.472226 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 573 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.472705 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 573 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.472669 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 573 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.472847 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 573 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.472848 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 573 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.473089 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 573 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 573: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:22.473460 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 573 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.473598 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 573 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 573 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 573: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:22.473635 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 573 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.473764 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 573 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 573 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.473824 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 573 election: Election decided. Result: candidate lost.
I1014 19:27:22.474810 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 573 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.475033 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 573: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.475392 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 573 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 573 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.475709 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 573 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 573: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.475770 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 573 election: Election decided. Result: candidate lost.
I1014 19:27:22.476054 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:22.476121 26735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 573 FOLLOWER]: Leader election lost for term 573. Reason: could not achieve majority
I1014 19:27:22.476238 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:22.476331 27337 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Leader election lost for term 573. Reason: could not achieve majority
I1014 19:27:22.477547 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 572. Current term is 573. Ops: []
I1014 19:27:22.478044 27451 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 572.406, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I1014 19:27:22.478298 26897 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Stepping down as leader of term 572
I1014 19:27:22.478350 26897 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:22.478543 26897 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 572.406, Last appended by leader: 406, Current term: 572, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.478724 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:22.478755 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:22.478862 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 572 FOLLOWER]: Advancing to term 573
I1014 19:27:22.561185 26897 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 573 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:22.561265 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:22.561322 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 573 FOLLOWER]: Advancing to term 574
I1014 19:27:22.562845 26897 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.563259 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 574 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.563388 26897 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 574 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.564054 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 574 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.564039 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 574 candidate_status { last_received { term: 572 index: 406 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.564188 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 573 FOLLOWER]: Advancing to term 574
I1014 19:27:22.564213 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 573 FOLLOWER]: Advancing to term 574
I1014 19:27:22.564261 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.209s (vote granted)
I1014 19:27:22.564282 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:27:22.565953 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 574.
I1014 19:27:22.566171 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 574 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 574.
I1014 19:27:22.566406 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 574 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.566460 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 574 election: Election decided. Result: candidate won.
I1014 19:27:22.566592 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 574 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.567072 26897 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:22.567137 26897 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 FOLLOWER]: Leader election won for term 574
I1014 19:27:22.567234 26897 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:22.567405 26897 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 572.406, Last appended by leader: 406, Current term: 574, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.568140 27451 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.568328 27451 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.568476 27446 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:22.569849 27446 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:22.570778 27446 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:22.571421 27446 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:22.573365 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 572 index: 406. Preceding OpId from leader: term: 574 index: 408. (index mismatch)
I1014 19:27:22.573468 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 574 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 572 index: 406. Preceding OpId from leader: term: 574 index: 408. (index mismatch)
I1014 19:27:22.573845 27451 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I1014 19:27:22.574026 26897 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I1014 19:27:22.575359 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.575577 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.575486 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.575727 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.576964 27451 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.577139 27451 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.577340 27451 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.577548 27451 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.577594 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.577777 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.578389 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 574 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.578673 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.601387 29304 heartbeater.cc:467] Master 127.20.201.60:53788 was elected leader, sending a full tablet report...
I1014 19:27:22.777149 27337 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 574 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:22.777225 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:22.777276 27337 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 574 FOLLOWER]: Advancing to term 575
I1014 19:27:22.779283 27337 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.779737 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 575 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.779872 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 575 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.783999 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 575 candidate_status { last_received { term: 574 index: 408 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.784238 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 575 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.784678 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 575 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 575 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.784930 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 575 candidate_status { last_received { term: 574 index: 408 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.785116 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 575 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.785565 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 575 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 575 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.785629 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 575 election: Election decided. Result: candidate lost.
I1014 19:27:22.785864 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:22.785929 27337 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Leader election lost for term 575. Reason: could not achieve majority
I1014 19:27:22.794456 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 574. Current term is 575. Ops: []
I1014 19:27:22.794934 26897 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 574.408, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I1014 19:27:22.795186 26897 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 LEADER]: Stepping down as leader of term 574
I1014 19:27:22.795224 26897 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:22.795404 26897 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 574.408, Last appended by leader: 408, Current term: 574, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.795629 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:22.795670 26897 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:22.795804 26897 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 574 FOLLOWER]: Advancing to term 575
I1014 19:27:22.842914 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:22.843044 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:22.843120 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 574 FOLLOWER]: Advancing to term 575
I1014 19:27:22.845295 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.845798 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 575 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.845955 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 575 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.846482 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 575 candidate_status { last_received { term: 574 index: 408 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.846669 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 575: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.846642 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 575 candidate_status { last_received { term: 574 index: 408 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:22.846799 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (vote granted)
I1014 19:27:22.847033 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 575 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 575: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:22.848786 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 575 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 575.
I1014 19:27:22.849275 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 575 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:22.849371 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 575 election: Election decided. Result: candidate won.
I1014 19:27:22.850188 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:22.850265 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 FOLLOWER]: Leader election won for term 575
I1014 19:27:22.850632 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:22.850879 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 574.408, Last appended by leader: 408, Current term: 575, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.852274 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.852643 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.857731 27634 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:22.859408 27634 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:22.860760 27634 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:22.862259 27634 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:22.864287 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 574 index: 408. Preceding OpId from leader: term: 575 index: 410. (index mismatch)
I1014 19:27:22.864434 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 575 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 574 index: 408. Preceding OpId from leader: term: 575 index: 410. (index mismatch)
I1014 19:27:22.864773 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I1014 19:27:22.865017 27633 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I1014 19:27:22.866510 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.866757 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.866641 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.866847 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.868233 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.868566 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.868736 25847 mvcc.cc:203] Tried to move safe_time back from 6306080533967818752 to 6306080533919649792. Current Snapshot: MvccSnapshot[committed={T|T < 6306080533967818752}]
I1014 19:27:22.869743 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.869906 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.875914 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.876513 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:22.877332 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 575 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:22.877573 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:22.970600 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 575 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:22.970687 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:22.970736 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 575 FOLLOWER]: Advancing to term 576
I1014 19:27:22.972880 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.973351 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 576 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:22.973500 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 576 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:22.974076 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 576 candidate_status { last_received { term: 575 index: 410 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:22.974267 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 576 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.974848 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 576 candidate_status { last_received { term: 575 index: 410 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:22.975042 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 576 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.975394 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 576 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 576 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.975539 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 576 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 576 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:22.975600 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 576 election: Election decided. Result: candidate lost.
I1014 19:27:22.976227 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:22.976318 27444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Leader election lost for term 576. Reason: could not achieve majority
I1014 19:27:22.988540 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 575. Current term is 576. Ops: []
I1014 19:27:22.989091 27636 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 575.410, Next index: 411, Last known committed idx: 410, Time since last communication: 0.000s
I1014 19:27:22.989387 27633 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 LEADER]: Stepping down as leader of term 575
I1014 19:27:22.989472 27633 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:22.989693 27633 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 410, Committed index: 410, Last appended: 575.410, Last appended by leader: 410, Current term: 575, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:22.989902 27633 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:22.989955 27633 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:22.990087 27633 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 575 FOLLOWER]: Advancing to term 576
I1014 19:27:23.063249 27337 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:23.063340 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:23.063427 27337 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 575 FOLLOWER]: Advancing to term 576
I1014 19:27:23.064867 27337 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.065264 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 576 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.065421 27337 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 576 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.065910 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 576 candidate_status { last_received { term: 575 index: 410 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.066056 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 576 candidate_status { last_received { term: 575 index: 410 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.066121 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 576: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.066207 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (vote granted)
I1014 19:27:23.067392 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 576 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 576: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.067821 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 576 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 576.
I1014 19:27:23.068095 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 576 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.068145 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 576 election: Election decided. Result: candidate won.
I1014 19:27:23.068469 27337 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.108s (election complete)
I1014 19:27:23.068553 27337 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 FOLLOWER]: Leader election won for term 576
I1014 19:27:23.068817 27337 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:23.068986 27337 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 410, Committed index: 410, Last appended: 575.410, Last appended by leader: 410, Current term: 576, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.069886 27696 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.070185 27696 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.070641 27697 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:23.072224 27697 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:23.073235 27697 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:23.073920 27697 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:23.076319 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 576 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 575 index: 410. Preceding OpId from leader: term: 576 index: 412. (index mismatch)
I1014 19:27:23.076694 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 575 index: 410. Preceding OpId from leader: term: 576 index: 412. (index mismatch)
I1014 19:27:23.077019 27696 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 411, Last known committed idx: 410, Time since last communication: 0.000s
I1014 19:27:23.077193 27337 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 411, Last known committed idx: 410, Time since last communication: 0.000s
I1014 19:27:23.079030 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.079058 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.079336 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.079342 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.081094 27337 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.081351 27337 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.081687 27696 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.082470 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.082705 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.081897 27696 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.083148 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 576 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.083343 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.182600 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:23.182684 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:23.182735 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 576 FOLLOWER]: Advancing to term 577
I1014 19:27:23.183485 27633 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 576 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:23.183578 27633 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:23.183645 27633 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 576 FOLLOWER]: Advancing to term 577
I1014 19:27:23.185000 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 577 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.185514 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 577 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.185652 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 577 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.185861 27633 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.186254 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 577 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.186359 27633 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 577 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.186424 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 577 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.186468 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 577 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.186677 29010 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 577 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:23.186821 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 577 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 577 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.187031 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 577 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Service unavailable: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 577 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:23.187065 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 576. Current term is 577. Ops: []
I1014 19:27:23.187086 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 577 election: Election decided. Result: candidate lost.
I1014 19:27:23.187247 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:23.187297 27444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 577 FOLLOWER]: Leader election lost for term 577. Reason: could not achieve majority
I1014 19:27:23.187309 27633 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 577 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.187564 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 577 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.187738 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 577 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 577: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.188061 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 577 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.188189 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 577 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.188400 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 577 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 577 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 577: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.188338 27699 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 576.412, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I1014 19:27:23.188642 27699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Stepping down as leader of term 576
I1014 19:27:23.188674 27699 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:23.188694 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 577 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 577 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.188766 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 577 election: Election decided. Result: candidate lost.
I1014 19:27:23.188822 27699 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 412, Committed index: 412, Last appended: 576.412, Last appended by leader: 412, Current term: 576, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.189014 27699 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:23.189100 27699 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:23.189111 27633 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:23.189177 27633 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Leader election lost for term 577. Reason: could not achieve majority
I1014 19:27:23.189201 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 576 FOLLOWER]: Advancing to term 577
I1014 19:27:23.295791 27633 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:23.295887 27633 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:23.295966 27633 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 577 FOLLOWER]: Advancing to term 578
I1014 19:27:23.305945 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 577 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:23.306035 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:23.306093 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 577 FOLLOWER]: Advancing to term 578
I1014 19:27:23.306366 27633 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.307971 27633 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 578 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.308121 27633 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 578 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.308225 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.308709 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 578 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.308849 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 578 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.309139 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 578 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.309136 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 578 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.309285 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 577 FOLLOWER]: Advancing to term 578
I1014 19:27:23.309345 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 578: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.309352 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (vote granted)
I1014 19:27:23.309641 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 578 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.309810 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 578: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.309815 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 578 candidate_status { last_received { term: 576 index: 412 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.310178 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 578 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 578: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.310355 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 578 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 578: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.311467 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 578.
I1014 19:27:23.311589 28639 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 578 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:23.311822 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 578 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.311884 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 578 election: Election decided. Result: candidate won.
I1014 19:27:23.312003 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 578 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 578 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:23.312074 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 578 election: Election decided. Result: candidate lost.
I1014 19:27:23.312268 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:23.312332 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Leader election lost for term 578. Reason: could not achieve majority
I1014 19:27:23.312355 27633 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:23.312412 27633 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 FOLLOWER]: Leader election won for term 578
I1014 19:27:23.312515 27633 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:23.312695 27633 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 412, Committed index: 412, Last appended: 576.412, Last appended by leader: 412, Current term: 578, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.313684 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.313868 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.314035 27634 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:23.315814 27634 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:23.317122 27634 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:23.317929 27634 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:23.321622 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 576 index: 412. Preceding OpId from leader: term: 578 index: 414. (index mismatch)
I1014 19:27:23.322238 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 576 index: 412. Preceding OpId from leader: term: 578 index: 414. (index mismatch)
I1014 19:27:23.322259 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I1014 19:27:23.322942 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 413, Last known committed idx: 412, Time since last communication: 0.000s
I1014 19:27:23.324183 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.324484 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.325165 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.325361 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.325567 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.325866 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.326251 25850 mvcc.cc:203] Tried to move safe_time back from 6306080535839887360 to 6306080535811235840. Current Snapshot: MvccSnapshot[committed={T|T < 6306080535839887360}]
I1014 19:27:23.326192 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.326349 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.326989 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.327155 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.327913 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 578 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.328114 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.426203 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:23.426282 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (starting election)
I1014 19:27:23.426331 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 578 FOLLOWER]: Advancing to term 579
I1014 19:27:23.428723 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 579 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.429250 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 579 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.429265 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:23.429327 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:23.429373 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 578 FOLLOWER]: Advancing to term 579
I1014 19:27:23.429424 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 579 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.429998 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 579 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.430303 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 579 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.430512 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 579 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.430878 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 579 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 579 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.431725 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.432034 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 579: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.432235 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 579 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.432364 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 579 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.432796 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 579 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 579: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.432855 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 579 election: Election decided. Result: candidate lost.
I1014 19:27:23.433040 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:23.433095 27444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 579 FOLLOWER]: Leader election lost for term 579. Reason: could not achieve majority
I1014 19:27:23.433329 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 578. Current term is 579. Ops: []
I1014 19:27:23.433542 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 579 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.433717 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 579 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 579: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.433883 27633 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 578.414, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I1014 19:27:23.433948 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 579 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.434132 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 579 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.434288 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 579 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 579 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 579: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:23.434392 26735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Stepping down as leader of term 578
I1014 19:27:23.434437 26735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:23.434698 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 579 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 579 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.434761 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 579 election: Election decided. Result: candidate lost.
I1014 19:27:23.434645 26735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 414, Committed index: 414, Last appended: 578.414, Last appended by leader: 414, Current term: 578, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.434825 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:23.434865 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:23.434926 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:23.434978 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Leader election lost for term 579. Reason: could not achieve majority
I1014 19:27:23.434998 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 578 FOLLOWER]: Advancing to term 579
I1014 19:27:23.545761 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 579 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:23.545838 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:23.545894 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 579 FOLLOWER]: Advancing to term 580
I1014 19:27:23.547580 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.547996 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 580 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.548158 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 580 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.548673 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:23.548738 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:23.548787 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 579 FOLLOWER]: Advancing to term 580
I1014 19:27:23.548777 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 580 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.548780 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 580 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.548928 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 579 FOLLOWER]: Advancing to term 580
I1014 19:27:23.548981 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.178s (vote granted)
I1014 19:27:23.550496 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.550783 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 580: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.550925 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 580 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.550961 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 580.
I1014 19:27:23.551049 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 580 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.551616 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 580 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.551573 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 580 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.551673 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 580 election: Election decided. Result: candidate won.
I1014 19:27:23.551729 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 580: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.551831 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:23.551885 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 FOLLOWER]: Leader election won for term 580
I1014 19:27:23.551878 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 580 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 580: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.551971 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:23.551946 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 580 candidate_status { last_received { term: 578 index: 414 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.552119 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 580: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.552155 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 414, Committed index: 414, Last appended: 578.414, Last appended by leader: 414, Current term: 580, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.552354 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 580 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 580: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.552697 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 580 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 580: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:23.552758 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 580 election: Election decided. Result: candidate lost.
I1014 19:27:23.553086 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.193s (election complete)
I1014 19:27:23.553145 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Leader election lost for term 580. Reason: could not achieve majority
I1014 19:27:23.553153 27633 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.553346 27633 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.553578 27634 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:23.554890 27634 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:23.555838 27634 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:23.556499 27634 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:23.558604 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 578 index: 414. Preceding OpId from leader: term: 580 index: 416. (index mismatch)
I1014 19:27:23.558934 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 578 index: 414. Preceding OpId from leader: term: 580 index: 416. (index mismatch)
I1014 19:27:23.559438 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I1014 19:27:23.559756 26735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 415, Last known committed idx: 414, Time since last communication: 0.000s
I1014 19:27:23.560961 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.561266 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.562443 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.562712 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.563050 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.563236 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.563653 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.563815 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.564689 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.564689 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 580 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.564867 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.564867 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.624148 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:23.664950 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:23.665030 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:23.665081 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 580 FOLLOWER]: Advancing to term 581
I1014 19:27:23.667191 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.667743 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 581 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:23.667897 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 581 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.668331 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 581 candidate_status { last_received { term: 580 index: 416 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:23.669247 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 580. Current term is 581. Ops: []
I1014 19:27:23.669209 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 581 candidate_status { last_received { term: 580 index: 416 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.669385 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 581 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.669772 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 581 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 581 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.669755 27636 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 580.416, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I1014 19:27:23.669986 26735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 LEADER]: Stepping down as leader of term 580
I1014 19:27:23.670024 26735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:23.670202 26735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 416, Committed index: 416, Last appended: 580.416, Last appended by leader: 416, Current term: 580, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.670383 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:23.670426 26735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:23.670586 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 580 FOLLOWER]: Advancing to term 581
I1014 19:27:23.678690 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 581 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.679122 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 581 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 581 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:23.679203 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 581 election: Election decided. Result: candidate lost.
I1014 19:27:23.679401 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:23.679488 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Leader election lost for term 581. Reason: could not achieve majority
I1014 19:27:23.760144 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:23.760272 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:23.760335 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 580 FOLLOWER]: Advancing to term 581
I1014 19:27:23.761976 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.762336 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 581 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:23.762490 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 581 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.762910 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 581 candidate_status { last_received { term: 580 index: 416 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:23.763062 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 581: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.763005 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 581 candidate_status { last_received { term: 580 index: 416 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:23.763159 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:23.763370 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 581 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 581: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:23.765142 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 581.
I1014 19:27:23.765450 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 581 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:23.765499 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 581 election: Election decided. Result: candidate won.
I1014 19:27:23.765661 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:23.765719 27444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 FOLLOWER]: Leader election won for term 581
I1014 19:27:23.765954 27444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:23.766191 27444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 416, Committed index: 416, Last appended: 580.416, Last appended by leader: 416, Current term: 581, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:23.766999 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.767199 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.767596 28011 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:23.769075 28011 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:23.770176 28011 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:23.770905 28011 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:23.773066 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 580 index: 416. Preceding OpId from leader: term: 581 index: 418. (index mismatch)
I1014 19:27:23.773066 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 580 index: 416. Preceding OpId from leader: term: 581 index: 418. (index mismatch)
I1014 19:27:23.773492 27444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I1014 19:27:23.773746 28005 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I1014 19:27:23.775130 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.775329 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.775609 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.775795 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.776850 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.777027 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.777915 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.778082 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:23.778578 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.778805 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:23.779070 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 581 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:23.779325 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.047192 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.047282 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:24.047325 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 581 FOLLOWER]: Advancing to term 582
I1014 19:27:24.049427 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.049739 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 581. Current term is 582. Ops: []
I1014 19:27:24.049945 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 582 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.050086 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 582 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.050604 27444 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 581.418, Next index: 419, Last known committed idx: 418, Time since last communication: 0.000s
I1014 19:27:24.050676 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 582 candidate_status { last_received { term: 581 index: 418 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.050868 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 582 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.051021 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 582 candidate_status { last_received { term: 581 index: 418 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.051203 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 582 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.051581 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 582 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 582 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.051653 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 582 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 582 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.051723 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 582 election: Election decided. Result: candidate lost.
I1014 19:27:24.051903 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:24.051951 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Leader election lost for term 582. Reason: could not achieve majority
I1014 19:27:24.051993 27444 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 LEADER]: Stepping down as leader of term 581
I1014 19:27:24.052036 27444 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:24.052222 27444 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 418, Committed index: 418, Last appended: 581.418, Last appended by leader: 418, Current term: 581, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.052399 27444 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:24.052433 27444 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:24.052587 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 581 FOLLOWER]: Advancing to term 582
I1014 19:27:24.058166 26735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.058254 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:24.058315 26735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 581 FOLLOWER]: Advancing to term 582
I1014 19:27:24.060421 26735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.060951 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 582 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.061111 26735 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 582 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.062144 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 582 candidate_status { last_received { term: 581 index: 418 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.062144 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 582 candidate_status { last_received { term: 581 index: 418 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.062307 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:24.062450 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 582: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:24.062810 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 582 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 582: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:24.064190 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 582 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 582.
I1014 19:27:24.064569 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 582 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.064635 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 582 election: Election decided. Result: candidate won.
I1014 19:27:24.065412 26735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:24.065493 26735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 FOLLOWER]: Leader election won for term 582
I1014 19:27:24.065635 26735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:24.065897 26735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 418, Committed index: 418, Last appended: 581.418, Last appended by leader: 418, Current term: 582, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.069173 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.069360 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.070009 28180 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.072211 28180 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.073518 28180 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.074741 28180 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:24.078470 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 581 index: 418. Preceding OpId from leader: term: 582 index: 420. (index mismatch)
I1014 19:27:24.078665 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 582 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 581 index: 418. Preceding OpId from leader: term: 582 index: 420. (index mismatch)
I1014 19:27:24.079154 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 419, Last known committed idx: 418, Time since last communication: 0.000s
I1014 19:27:24.079542 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 419, Last known committed idx: 418, Time since last communication: 0.000s
I1014 19:27:24.081634 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.081828 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.082084 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.082271 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.082592 26735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.082788 26735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.084370 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.084561 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.087342 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.087553 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.088860 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 582 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.089031 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.183962 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 582 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:24.184079 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:24.184131 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 582 FOLLOWER]: Advancing to term 583
I1014 19:27:24.186326 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.186925 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 583 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.187129 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 583 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.187861 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 583 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.188066 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.188040 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 583 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.188256 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.188576 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:24.188639 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:24.188694 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Advancing to term 583
I1014 19:27:24.188879 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 583 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.189169 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 583 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 582 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.189224 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 583 election: Election decided. Result: candidate lost.
I1014 19:27:24.189551 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:24.189602 27444 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Leader election lost for term 583. Reason: could not achieve majority
I1014 19:27:24.190878 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 583 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.191438 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 583 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.191658 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 583 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.192209 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 583 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.192394 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 583: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:24.193023 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 583 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.193202 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.193553 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 583 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 583 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.193862 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 582. Current term is 583. Ops: []
I1014 19:27:24.194180 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 583 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 583: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:24.194244 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 583 election: Election decided. Result: candidate lost.
I1014 19:27:24.194453 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:24.194548 27699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 583 FOLLOWER]: Leader election lost for term 583. Reason: could not achieve majority
I1014 19:27:24.194813 26735 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 582.420, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I1014 19:27:24.195015 27636 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Stepping down as leader of term 582
I1014 19:27:24.195057 27636 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:24.195225 27636 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 582.420, Last appended by leader: 420, Current term: 582, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.195358 27636 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:24.195396 27636 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:24.195531 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 582 FOLLOWER]: Advancing to term 583
I1014 19:27:24.297808 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:24.297870 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.166s (starting election)
I1014 19:27:24.297915 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 583 FOLLOWER]: Advancing to term 584
I1014 19:27:24.299877 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.300252 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 584 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.300390 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 584 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.300837 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 584 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.300945 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 583 FOLLOWER]: Advancing to term 584
I1014 19:27:24.300992 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.200s (vote granted)
I1014 19:27:24.300961 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 584 candidate_status { last_received { term: 582 index: 420 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.301110 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 583 FOLLOWER]: Advancing to term 584
I1014 19:27:24.301179 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (vote granted)
I1014 19:27:24.302635 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 584.
I1014 19:27:24.302963 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 584 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 584.
I1014 19:27:24.302989 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 584 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.303035 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 584 election: Election decided. Result: candidate won.
I1014 19:27:24.303321 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 584 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.303530 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:24.303586 27444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 FOLLOWER]: Leader election won for term 584
I1014 19:27:24.303668 27444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:24.303822 27444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 582.420, Last appended by leader: 420, Current term: 584, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.304556 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.304739 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.305136 28274 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.306394 28274 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.307894 28274 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.308511 28274 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:24.310115 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 584 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 582 index: 420. Preceding OpId from leader: term: 584 index: 422. (index mismatch)
I1014 19:27:24.310261 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 582 index: 420. Preceding OpId from leader: term: 584 index: 422. (index mismatch)
I1014 19:27:24.310555 27444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I1014 19:27:24.310696 28005 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.000s
I1014 19:27:24.312113 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.312288 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.312430 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.312598 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.313592 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.313757 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.315557 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.315721 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.316570 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.316738 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.317286 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 584 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.317457 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.415567 27636 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 584 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.415647 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:24.415709 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 584 FOLLOWER]: Advancing to term 585
I1014 19:27:24.418113 27636 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.418630 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 585 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.418772 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 585 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.420135 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 585 candidate_status { last_received { term: 584 index: 422 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.420320 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 585 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.420444 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 585 candidate_status { last_received { term: 584 index: 422 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.420651 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 585 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.421010 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 585 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 585 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.421139 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 585 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 585 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.421185 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 585 election: Election decided. Result: candidate lost.
I1014 19:27:24.421494 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:24.421561 27636 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Leader election lost for term 585. Reason: could not achieve majority
I1014 19:27:24.434763 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 584. Current term is 585. Ops: []
I1014 19:27:24.435281 28276 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 584.422, Next index: 423, Last known committed idx: 422, Time since last communication: 0.000s
I1014 19:27:24.435561 28276 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 LEADER]: Stepping down as leader of term 584
I1014 19:27:24.435596 28276 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:24.435770 28276 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 584.422, Last appended by leader: 422, Current term: 584, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.435937 28276 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:24.435967 28276 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:24.436071 28276 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 584 FOLLOWER]: Advancing to term 585
I1014 19:27:24.509076 27699 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.509161 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:24.509212 27699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 584 FOLLOWER]: Advancing to term 585
I1014 19:27:24.510692 27699 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.511196 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 585 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.511369 27699 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 585 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.511924 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 585 candidate_status { last_received { term: 584 index: 422 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.512092 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:24.512106 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 585 candidate_status { last_received { term: 584 index: 422 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.512289 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 585: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.512797 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 585 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 585: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.514189 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 585.
I1014 19:27:24.514516 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 585 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.514571 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 585 election: Election decided. Result: candidate won.
I1014 19:27:24.514761 27699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:24.514818 27699 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 FOLLOWER]: Leader election won for term 585
I1014 19:27:24.515112 27699 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:24.515287 27699 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 584.422, Last appended by leader: 422, Current term: 585, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.516232 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.516520 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.516990 28318 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.520323 28318 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.521737 28318 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.522620 28318 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:24.525338 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 584 index: 422. Preceding OpId from leader: term: 585 index: 424. (index mismatch)
I1014 19:27:24.525338 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 584 index: 422. Preceding OpId from leader: term: 585 index: 424. (index mismatch)
I1014 19:27:24.525897 27699 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 422, Time since last communication: 0.000s
I1014 19:27:24.526127 28317 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 422, Time since last communication: 0.000s
I1014 19:27:24.527885 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.527881 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.528084 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.528092 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.530498 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.530763 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.532812 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.533001 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.536068 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.536317 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.537094 27444 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 585 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.537288 27444 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.606148 29304 heartbeater.cc:467] Master 127.20.201.60:53788 was elected leader, sending a full tablet report...
I1014 19:27:24.632872 27636 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:24.632972 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:24.633026 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 585 FOLLOWER]: Advancing to term 586
I1014 19:27:24.634702 27636 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.635277 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 586 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.635504 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 586 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.636158 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 586 candidate_status { last_received { term: 585 index: 424 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.636174 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 586 candidate_status { last_received { term: 585 index: 424 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.636350 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 586 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.636364 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 586 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.636770 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 586 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 586 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.636955 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 586 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 586 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.637014 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 586 election: Election decided. Result: candidate lost.
I1014 19:27:24.637465 27444 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:24.637533 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 585. Current term is 586. Ops: []
I1014 19:27:24.637548 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:24.637603 27444 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 585 FOLLOWER]: Advancing to term 586
I1014 19:27:24.637897 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:24.637954 27636 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Leader election lost for term 586. Reason: could not achieve majority
I1014 19:27:24.638062 27699 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 585.424, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I1014 19:27:24.638268 28317 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 LEADER]: Stepping down as leader of term 585
I1014 19:27:24.638308 28317 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:24.638484 28317 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 585.424, Last appended by leader: 424, Current term: 585, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.638643 28317 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:24.638674 28317 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:24.638774 28317 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 585 FOLLOWER]: Advancing to term 586
I1014 19:27:24.639686 27444 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.640166 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 586 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.640310 27444 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 586 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.640846 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 586 candidate_status { last_received { term: 585 index: 424 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.640847 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 586 candidate_status { last_received { term: 585 index: 424 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.640965 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:24.641005 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 586: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.641290 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 586 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 586: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.642565 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 586 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 586.
I1014 19:27:24.642877 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 586 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.642937 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 586 election: Election decided. Result: candidate won.
I1014 19:27:24.643257 27444 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:24.643321 27444 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 FOLLOWER]: Leader election won for term 586
I1014 19:27:24.643553 27444 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:24.643750 27444 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 585.424, Last appended by leader: 424, Current term: 586, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.645123 28276 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.645308 28276 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.645519 28274 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.647207 28274 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.648402 28274 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.649158 28274 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:24.651528 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 585 index: 424. Preceding OpId from leader: term: 586 index: 426. (index mismatch)
I1014 19:27:24.651690 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 586 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 585 index: 424. Preceding OpId from leader: term: 586 index: 426. (index mismatch)
I1014 19:27:24.652282 27444 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I1014 19:27:24.652495 28276 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I1014 19:27:24.653936 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.654146 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.654290 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.654484 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.655889 28005 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.656080 28005 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.657352 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.657676 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.657858 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.658025 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.658340 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 586 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.658536 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.757478 27636 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.757563 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:24.757624 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 586 FOLLOWER]: Advancing to term 587
I1014 19:27:24.757987 28317 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 586 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:24.758060 28317 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:24.758101 28317 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 586 FOLLOWER]: Advancing to term 587
I1014 19:27:24.759744 27636 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.760054 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 586. Current term is 587. Ops: []
I1014 19:27:24.760072 28317 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.760275 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 587 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.760483 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 587 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.760535 28317 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 587 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.760674 28317 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 587 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.760969 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 587 candidate_status { last_received { term: 586 index: 426 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.761149 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 587 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.761227 27444 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 586.426, Next index: 427, Last known committed idx: 426, Time since last communication: 0.000s
I1014 19:27:24.761291 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 587 candidate_status { last_received { term: 586 index: 426 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.761489 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 587: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.761447 28341 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 LEADER]: Stepping down as leader of term 586
I1014 19:27:24.761528 28341 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:24.761580 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 587 candidate_status { last_received { term: 586 index: 426 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.761751 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 587: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:24.761724 28341 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 426, Committed index: 426, Last appended: 586.426, Last appended by leader: 426, Current term: 586, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.761862 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 587 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 587: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:24.761917 28341 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:24.761960 28341 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:24.762089 28341 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 FOLLOWER]: Advancing to term 587
I1014 19:27:24.762267 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 587 candidate_status { last_received { term: 586 index: 426 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.764212 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:24.764420 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 587 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 586 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 587 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.764583 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 587 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 587: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:24.764632 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 587 election: Election decided. Result: candidate lost.
I1014 19:27:24.764912 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:24.764958 27636 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election lost for term 587. Reason: could not achieve majority
I1014 19:27:24.765955 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 587 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 587.
I1014 19:27:24.766692 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 587 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.766752 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 587 election: Election decided. Result: candidate won.
I1014 19:27:24.766933 28317 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:24.766993 28317 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 FOLLOWER]: Leader election won for term 587
I1014 19:27:24.767081 28317 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:24.767271 28317 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 426, Committed index: 426, Last appended: 586.426, Last appended by leader: 426, Current term: 587, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.768541 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.768716 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.768896 28318 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.770786 28318 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.772022 28318 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.772874 28318 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:24.775297 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 586 index: 426. Preceding OpId from leader: term: 587 index: 428. (index mismatch)
I1014 19:27:24.775379 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 587 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 586 index: 426. Preceding OpId from leader: term: 587 index: 428. (index mismatch)
I1014 19:27:24.775877 28317 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 427, Last known committed idx: 426, Time since last communication: 0.000s
I1014 19:27:24.777509 28317 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 427, Last known committed idx: 426, Time since last communication: 0.000s
I1014 19:27:24.777825 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.778025 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.778856 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.779039 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.779273 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.779461 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.780195 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.780376 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.780903 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.781087 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.781556 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 587 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.781728 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.881701 28341 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 587 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:24.881783 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:24.881836 28341 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 587 FOLLOWER]: Advancing to term 588
I1014 19:27:24.884042 28341 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 588 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.884548 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 588 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.884698 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 588 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.885376 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 588 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.885591 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 588 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.885637 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 588 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.885812 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 588 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.886574 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 588 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 588 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.886848 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 588 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 588 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.886903 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 588 election: Election decided. Result: candidate lost.
I1014 19:27:24.887075 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:24.887130 28341 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 588 FOLLOWER]: Leader election lost for term 588. Reason: could not achieve majority
I1014 19:27:24.889686 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 588 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 587. Current term is 588. Ops: []
I1014 19:27:24.890127 28317 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 587.428, Next index: 429, Last known committed idx: 428, Time since last communication: 0.000s
I1014 19:27:24.890332 28317 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 LEADER]: Stepping down as leader of term 587
I1014 19:27:24.890365 28317 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:24.890539 28317 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 428, Committed index: 428, Last appended: 587.428, Last appended by leader: 428, Current term: 587, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.890702 28317 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:24.890731 28317 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:24.890830 28317 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 587 FOLLOWER]: Advancing to term 588
I1014 19:27:24.969565 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 588 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:24.969650 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:24.969705 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 588 FOLLOWER]: Advancing to term 589
I1014 19:27:24.971434 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.971962 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 589 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.972111 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 589 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:24.972573 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 589 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.972708 28639 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 588 FOLLOWER]: Advancing to term 589
I1014 19:27:24.972771 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:24.973209 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 589 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:24.973425 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 589 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.973819 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 589 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 589 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.974694 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 589.
I1014 19:27:24.974983 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 589 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.975045 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 589 election: Election decided. Result: candidate won.
I1014 19:27:24.975239 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:24.975298 28319 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 FOLLOWER]: Leader election won for term 589
I1014 19:27:24.975383 28319 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:24.975744 27636 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:24.975805 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:24.975860 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 587 FOLLOWER]: Advancing to term 588
I1014 19:27:24.975594 28319 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 428, Committed index: 428, Last appended: 587.428, Last appended by leader: 428, Current term: 589, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.977114 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.977288 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.977484 28318 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:24.977578 27636 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 588 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:24.977931 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 588 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:24.978063 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 588 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:24.978554 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 588 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:24.978687 28639 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 588. Current term is 589.
W1014 19:27:24.978982 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 588 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 588. Current term is 589.
I1014 19:27:24.979013 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 588 election: Cancelling election due to peer responding with higher term
I1014 19:27:24.979169 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:24.979221 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 588 FOLLOWER]: Advancing to term 589
I1014 19:27:24.979357 28318 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:24.979465 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 588 candidate_status { last_received { term: 587 index: 428 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:24.979647 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 588 because replica is either leader or believes a valid leader to be alive.
W1014 19:27:24.980013 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 588 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 588 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:24.980782 28318 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:24.980773 27636 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Leader election lost for term 588. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 588. Current term is 589.
I1014 19:27:24.981726 28318 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:24.983912 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 587 index: 428. Preceding OpId from leader: term: 589 index: 430. (index mismatch)
I1014 19:27:24.984225 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 587 index: 428. Preceding OpId from leader: term: 589 index: 430. (index mismatch)
I1014 19:27:24.984376 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 429, Last known committed idx: 428, Time since last communication: 0.000s
I1014 19:27:24.984699 28317 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 429, Last known committed idx: 428, Time since last communication: 0.000s
I1014 19:27:24.985878 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.986140 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.986220 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.986352 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.988384 27699 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.988586 27699 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.988876 28317 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.989042 28317 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:24.989181 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.989593 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:24.990231 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 589 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:24.990376 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.089373 28341 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:25.089458 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:25.089534 28341 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 589 FOLLOWER]: Advancing to term 590
I1014 19:27:25.091671 28341 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.092131 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 590 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.092275 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 590 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.093065 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 590 candidate_status { last_received { term: 589 index: 430 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.093268 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 590 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.093675 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 590 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 590 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.093978 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 590 candidate_status { last_received { term: 589 index: 430 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:25.094156 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 590 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.094656 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 590 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 590 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.094722 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 590 election: Election decided. Result: candidate lost.
I1014 19:27:25.094894 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:25.094952 28341 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Leader election lost for term 590. Reason: could not achieve majority
I1014 19:27:25.095268 28639 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 589. Current term is 590. Ops: []
I1014 19:27:25.095865 27699 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 589.430, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I1014 19:27:25.096088 28319 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Stepping down as leader of term 589
I1014 19:27:25.096127 28319 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:25.096290 28319 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 589.430, Last appended by leader: 430, Current term: 589, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.096505 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:25.096545 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:25.096652 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 589 FOLLOWER]: Advancing to term 590
I1014 19:27:25.179910 27636 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:25.180016 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:25.180074 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 589 FOLLOWER]: Advancing to term 590
I1014 19:27:25.182332 27636 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.182838 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 590 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.182998 27636 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 590 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.183524 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 590 candidate_status { last_received { term: 589 index: 430 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:25.183537 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 590 candidate_status { last_received { term: 589 index: 430 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:25.183717 28639 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 590: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:25.183691 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (vote granted)
I1014 19:27:25.185540 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 590 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 590: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:25.185716 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 590 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 590.
I1014 19:27:25.185997 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 590 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.186048 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 590 election: Election decided. Result: candidate won.
I1014 19:27:25.186218 27636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:27:25.186275 27636 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 FOLLOWER]: Leader election won for term 590
I1014 19:27:25.186573 27636 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:25.186800 27636 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 589.430, Last appended by leader: 430, Current term: 590, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.188468 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.189669 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.190147 28529 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:25.191568 28529 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:25.192525 28529 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:25.193181 28529 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:25.195003 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 590 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 589 index: 430. Preceding OpId from leader: term: 590 index: 432. (index mismatch)
I1014 19:27:25.195199 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 589 index: 430. Preceding OpId from leader: term: 590 index: 432. (index mismatch)
I1014 19:27:25.195546 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I1014 19:27:25.195792 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I1014 19:27:25.197388 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.197669 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.197814 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.197999 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.199617 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.199786 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.200234 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.200404 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.200227 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.200635 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.201002 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 590 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.201283 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.299471 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 590 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:25.299576 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:25.299625 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 590 FOLLOWER]: Advancing to term 591
I1014 19:27:25.301612 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.302027 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 591 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.302428 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 591 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.302811 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 591 candidate_status { last_received { term: 590 index: 432 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:25.302978 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 591 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.302961 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 591 candidate_status { last_received { term: 590 index: 432 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.303141 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 591 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.303359 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 591 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 591 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.303534 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 591 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 591 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.303596 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 591 election: Election decided. Result: candidate lost.
I1014 19:27:25.303838 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:25.303894 28319 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Leader election lost for term 591. Reason: could not achieve majority
I1014 19:27:25.309495 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 590. Current term is 591. Ops: []
I1014 19:27:25.310020 27636 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 590.432, Next index: 433, Last known committed idx: 432, Time since last communication: 0.000s
I1014 19:27:25.310235 27636 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 LEADER]: Stepping down as leader of term 590
I1014 19:27:25.310268 27636 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:25.310432 27636 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 432, Committed index: 432, Last appended: 590.432, Last appended by leader: 432, Current term: 590, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.310693 27636 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:25.310736 27636 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:25.310871 27636 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 590 FOLLOWER]: Advancing to term 591
I1014 19:27:25.382066 28341 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:25.382144 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:25.382197 28341 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 590 FOLLOWER]: Advancing to term 591
I1014 19:27:25.383661 28341 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.384078 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 591 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.384222 28341 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 591 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.384809 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 591 candidate_status { last_received { term: 590 index: 432 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:25.384840 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 591 candidate_status { last_received { term: 590 index: 432 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.385025 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 591: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.385013 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:25.385609 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 591 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 591: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.387146 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 591.
I1014 19:27:25.387457 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 591 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.387506 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 591 election: Election decided. Result: candidate won.
I1014 19:27:25.387687 28341 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:25.387745 28341 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 FOLLOWER]: Leader election won for term 591
I1014 19:27:25.388036 28341 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:25.388202 28341 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 432, Committed index: 432, Last appended: 590.432, Last appended by leader: 432, Current term: 591, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.388965 28683 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.389216 28683 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.389605 28684 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:25.391079 28684 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:25.392066 28684 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:25.392794 28684 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:25.394683 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 590 index: 432. Preceding OpId from leader: term: 591 index: 434. (index mismatch)
I1014 19:27:25.394743 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 590 index: 432. Preceding OpId from leader: term: 591 index: 434. (index mismatch)
I1014 19:27:25.395233 28341 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 433, Last known committed idx: 432, Time since last communication: 0.000s
I1014 19:27:25.395390 28683 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 433, Last known committed idx: 432, Time since last communication: 0.000s
I1014 19:27:25.397043 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.397231 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.396931 28531 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.397301 28531 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.398933 28341 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.399111 28341 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.399070 28683 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.399221 28683 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.399222 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.399385 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.401141 28531 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 591 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.401353 28531 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.589192 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:25.589299 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:25.589366 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 591 FOLLOWER]: Advancing to term 592
I1014 19:27:25.591477 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.592000 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 592 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.592219 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 592 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.592741 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 592 candidate_status { last_received { term: 591 index: 434 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:25.592921 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 592 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.592849 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 592 candidate_status { last_received { term: 591 index: 434 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.593039 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 592 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.593266 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 592 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 592 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.593447 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 592 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 592 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.593550 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 592 election: Election decided. Result: candidate lost.
I1014 19:27:25.593785 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:25.593852 28319 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Leader election lost for term 592. Reason: could not achieve majority
I1014 19:27:25.612318 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 591. Current term is 592. Ops: []
I1014 19:27:25.612761 28683 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 591.434, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I1014 19:27:25.613019 28685 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 LEADER]: Stepping down as leader of term 591
I1014 19:27:25.613061 28685 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:25.613235 28685 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 591.434, Last appended by leader: 434, Current term: 591, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.613463 28685 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:25.613514 28685 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:25.613633 28685 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 591 FOLLOWER]: Advancing to term 592
I1014 19:27:25.621503 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
I1014 19:27:25.676575 28531 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:25.676656 28531 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:25.676708 28531 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 591 FOLLOWER]: Advancing to term 592
I1014 19:27:25.678191 28531 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.678637 28531 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 592 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.678783 28531 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 592 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.679245 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 592 candidate_status { last_received { term: 591 index: 434 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:25.679389 28639 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:25.679596 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 592 candidate_status { last_received { term: 591 index: 434 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:25.679786 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 592: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.680105 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 592 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 592: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.680845 28639 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 592.
I1014 19:27:25.681118 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 592 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.681166 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 592 election: Election decided. Result: candidate won.
I1014 19:27:25.681357 28531 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:25.681412 28531 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 FOLLOWER]: Leader election won for term 592
I1014 19:27:25.681511 28531 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:25.681725 28531 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 591.434, Last appended by leader: 434, Current term: 592, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.682657 27636 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.682862 27636 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.683020 28529 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:25.684453 28529 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:25.685475 28529 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:25.686094 28529 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:25.687919 28639 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 591 index: 434. Preceding OpId from leader: term: 592 index: 436. (index mismatch)
I1014 19:27:25.687990 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 591 index: 434. Preceding OpId from leader: term: 592 index: 436. (index mismatch)
I1014 19:27:25.688343 28531 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I1014 19:27:25.688560 27636 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I1014 19:27:25.690110 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.690091 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.690305 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.690340 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.691874 28532 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.692251 28532 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.692446 28532 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.692622 28532 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.693336 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.693526 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.693801 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 592 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.694108 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.892220 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:25.892323 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:25.892397 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 592 FOLLOWER]: Advancing to term 593
I1014 19:27:25.894651 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.895150 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 593 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:25.895365 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 593 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.895865 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 593 candidate_status { last_received { term: 592 index: 436 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:25.895946 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 593 candidate_status { last_received { term: 592 index: 436 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.896071 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 593 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.896160 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 593 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.896564 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 593 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 593 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.896683 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 593 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 593 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:25.896769 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 593 election: Election decided. Result: candidate lost.
I1014 19:27:25.897068 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:25.897148 28319 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election lost for term 593. Reason: could not achieve majority
I1014 19:27:25.900952 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 592. Current term is 593. Ops: []
I1014 19:27:25.901377 28528 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 592.436, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I1014 19:27:25.901695 28528 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 LEADER]: Stepping down as leader of term 592
I1014 19:27:25.901736 28528 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:25.901970 28528 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 436, Committed index: 436, Last appended: 592.436, Last appended by leader: 436, Current term: 592, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.902158 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:25.902197 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:25.902326 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 592 FOLLOWER]: Advancing to term 593
I1014 19:27:25.968322 28685 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:25.968435 28685 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:25.968513 28685 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 592 FOLLOWER]: Advancing to term 593
I1014 19:27:25.970064 28685 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.970592 28685 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 593 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:25.970795 28685 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 593 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.971289 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 593 candidate_status { last_received { term: 592 index: 436 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:25.971504 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 593: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.971606 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 593 candidate_status { last_received { term: 592 index: 436 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:25.971771 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (vote granted)
I1014 19:27:25.971941 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 593 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 593: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:25.973935 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 593 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 593.
I1014 19:27:25.974634 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 593 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:25.974707 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 593 election: Election decided. Result: candidate won.
I1014 19:27:25.975282 28685 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:25.975370 28685 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 FOLLOWER]: Leader election won for term 593
I1014 19:27:25.975553 28685 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:25.975767 28685 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 436, Committed index: 436, Last appended: 592.436, Last appended by leader: 436, Current term: 593, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:25.977084 28683 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.977288 28683 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.977916 28969 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:25.980062 28969 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:25.981530 28969 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:25.982502 28969 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:25.985157 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 592 index: 436. Preceding OpId from leader: term: 593 index: 438. (index mismatch)
I1014 19:27:25.985700 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 593 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 592 index: 436. Preceding OpId from leader: term: 593 index: 438. (index mismatch)
I1014 19:27:25.985702 28685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I1014 19:27:25.986701 28685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 437, Last known committed idx: 436, Time since last communication: 0.000s
I1014 19:27:25.988118 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.988333 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.988219 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.988461 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.989547 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.989848 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:25.990267 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.990451 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.995529 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.996987 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 593 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:25.997189 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:25.997647 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.172646 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 593 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.172731 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:26.172793 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 593 FOLLOWER]: Advancing to term 594
I1014 19:27:26.175590 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 594 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.176139 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 594 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.176759 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 594 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.176983 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.177307 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 594 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.176311 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 594 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.177974 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 594 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.178177 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.178560 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 594 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.178624 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 594 election: Election decided. Result: candidate lost.
I1014 19:27:26.178915 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:26.179497 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 594 FOLLOWER]: Leader election lost for term 594. Reason: could not achieve majority
I1014 19:27:26.180824 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 594 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 593. Current term is 594. Ops: []
I1014 19:27:26.181308 28685 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 593.438, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I1014 19:27:26.181561 28971 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 LEADER]: Stepping down as leader of term 593
I1014 19:27:26.181639 28971 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:26.181896 28971 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 593.438, Last appended by leader: 438, Current term: 593, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.182181 28971 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:26.182255 28971 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:26.182400 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 593 FOLLOWER]: Advancing to term 594
I1014 19:27:26.268708 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 594 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:26.268818 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:26.268890 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 594 FOLLOWER]: Advancing to term 595
I1014 19:27:26.271126 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.271713 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 595 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.272462 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 595 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.272591 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 595 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.272792 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 595 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.273134 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 595 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:26.273210 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 595 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 595 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.273289 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 594 FOLLOWER]: Advancing to term 595
I1014 19:27:26.273365 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (vote granted)
I1014 19:27:26.275671 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 595.
I1014 19:27:26.276095 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 595 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.276162 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 595 election: Election decided. Result: candidate won.
I1014 19:27:26.276536 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:26.276610 28971 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 FOLLOWER]: Leader election won for term 595
I1014 19:27:26.276710 28971 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:26.276914 28971 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 593.438, Last appended by leader: 438, Current term: 595, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.277985 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.278182 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.278424 28969 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:26.280550 28969 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:26.281038 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.281124 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:26.281193 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 593 FOLLOWER]: Advancing to term 594
I1014 19:27:26.282117 28969 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:26.283008 28969 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:26.283335 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 594 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.283881 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 594 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.284093 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 594 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.284984 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 594 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:26.285169 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 594. Current term is 595.
W1014 19:27:26.285642 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 594 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 594. Current term is 595.
I1014 19:27:26.285599 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 594 candidate_status { last_received { term: 593 index: 438 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.285684 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 594 election: Cancelling election due to peer responding with higher term
I1014 19:27:26.285768 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.286022 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 594 FOLLOWER]: Advancing to term 595
W1014 19:27:26.286356 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 594 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 594 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.288226 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 595 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 593 index: 438. Preceding OpId from leader: term: 595 index: 440. (index mismatch)
I1014 19:27:26.288585 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:26.288862 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 593 index: 438. Preceding OpId from leader: term: 595 index: 440. (index mismatch)
I1014 19:27:26.288951 28319 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 595 FOLLOWER]: Leader election lost for term 594. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 594. Current term is 595.
I1014 19:27:26.289021 28971 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I1014 19:27:26.289508 28685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I1014 19:27:26.292331 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.292565 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.292479 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.292721 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.293700 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.294039 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.294317 28683 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.294493 28683 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.295469 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.295713 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.296330 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 595 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.296602 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.395530 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 595 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.395669 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:26.395768 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 595 FOLLOWER]: Advancing to term 596
I1014 19:27:26.396876 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.396939 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:26.396978 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 595 FOLLOWER]: Advancing to term 596
I1014 19:27:26.397858 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.398429 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 596 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.398631 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 596 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.399020 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.399262 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 595. Current term is 596. Ops: []
I1014 19:27:26.399524 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 596 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.399557 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 596 candidate_status { last_received { term: 595 index: 440 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:26.399775 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 596: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.399665 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 596 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.400202 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 596 candidate_status { last_received { term: 595 index: 440 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.400439 28639 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 596 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.400557 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 596 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 596: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.400537 28971 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 595.440, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I1014 19:27:26.400746 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 596 candidate_status { last_received { term: 595 index: 440 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.400822 28971 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Stepping down as leader of term 595
I1014 19:27:26.400923 28971 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:26.401146 28971 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 595.440, Last appended by leader: 440, Current term: 595, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.401276 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 596 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 596 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.401330 28971 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:26.401393 28971 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:26.401398 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 596 candidate_status { last_received { term: 595 index: 440 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.401582 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 596: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:26.401571 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 595 FOLLOWER]: Advancing to term 596
I1014 19:27:26.401880 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 596 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 596: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:26.401937 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 596 election: Election decided. Result: candidate lost.
I1014 19:27:26.402108 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:26.402160 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Leader election lost for term 596. Reason: could not achieve majority
I1014 19:27:26.403633 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (vote granted)
I1014 19:27:26.405668 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 596.
I1014 19:27:26.406240 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 596 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.406314 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 596 election: Election decided. Result: candidate won.
I1014 19:27:26.406635 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:26.406713 28319 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Leader election won for term 596
I1014 19:27:26.414674 28319 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:26.415225 28319 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 595.440, Last appended by leader: 440, Current term: 596, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.417527 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.417780 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.425983 29156 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:26.427899 29156 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:26.429303 29156 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:26.430169 29156 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:26.432337 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 595 index: 440. Preceding OpId from leader: term: 596 index: 442. (index mismatch)
I1014 19:27:26.432386 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 595 index: 440. Preceding OpId from leader: term: 596 index: 442. (index mismatch)
I1014 19:27:26.432890 29154 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I1014 19:27:26.433101 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I1014 19:27:26.434530 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.434774 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.434921 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.435149 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.435581 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.435778 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.436969 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.437149 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.437919 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.438166 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.438053 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 596 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.438382 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.539064 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:26.539144 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:26.539197 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 596 FOLLOWER]: Advancing to term 597
I1014 19:27:26.541352 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.541883 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 597 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.542024 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 597 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.544442 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 597 candidate_status { last_received { term: 596 index: 442 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.544672 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 597 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.544945 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 597 candidate_status { last_received { term: 596 index: 442 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.545150 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 597 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.545521 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 597 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 597 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.545656 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 597 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 597 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.545708 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 597 election: Election decided. Result: candidate lost.
I1014 19:27:26.545894 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:26.545949 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Leader election lost for term 597. Reason: could not achieve majority
I1014 19:27:26.548599 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 596. Current term is 597. Ops: []
I1014 19:27:26.549080 29154 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 596.442, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I1014 19:27:26.549371 29154 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 LEADER]: Stepping down as leader of term 596
I1014 19:27:26.549432 29154 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:26.549657 29154 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 596.442, Last appended by leader: 442, Current term: 596, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.549849 29154 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:26.549904 29154 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:26.550024 29154 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 596 FOLLOWER]: Advancing to term 597
I1014 19:27:26.623742 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:26.623845 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:26.623924 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 596 FOLLOWER]: Advancing to term 597
I1014 19:27:26.626595 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.627116 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 597 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.627321 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 597 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.627723 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 597 candidate_status { last_received { term: 596 index: 442 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.627887 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (vote granted)
I1014 19:27:26.627907 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 597 candidate_status { last_received { term: 596 index: 442 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:26.628100 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 597: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.628459 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 597 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 597: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.629444 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 597.
I1014 19:27:26.629755 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 597 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.629817 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 597 election: Election decided. Result: candidate won.
I1014 19:27:26.630071 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:26.630146 28971 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 FOLLOWER]: Leader election won for term 597
I1014 19:27:26.630280 28971 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:26.630481 28971 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 596.442, Last appended by leader: 442, Current term: 597, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.631613 28685 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.631814 28685 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.632021 28969 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:26.633535 28969 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:26.634620 28969 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:26.635282 28969 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:26.636986 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 596 index: 442. Preceding OpId from leader: term: 597 index: 444. (index mismatch)
I1014 19:27:26.637033 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 596 index: 442. Preceding OpId from leader: term: 597 index: 444. (index mismatch)
I1014 19:27:26.637442 28971 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I1014 19:27:26.637712 28685 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I1014 19:27:26.639124 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.639331 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.639677 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.639889 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.641710 28683 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.641934 28683 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.642076 25848 mvcc.cc:203] Tried to move safe_time back from 6306080549421162496 to 6306080549401624576. Current Snapshot: MvccSnapshot[committed={T|T < 6306080549421162496}]
I1014 19:27:26.642109 28970 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.642393 28970 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.642241 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.642534 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.642632 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 597 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.642817 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.742696 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.742784 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:26.742839 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 597 FOLLOWER]: Advancing to term 598
I1014 19:27:26.744428 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.744877 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 598 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.745030 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 598 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.745640 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 598 candidate_status { last_received { term: 597 index: 444 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.745751 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 598 candidate_status { last_received { term: 597 index: 444 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.745831 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 598 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.745883 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 598 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.746179 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 598 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 598 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.746314 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 598 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 598 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.746364 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 598 election: Election decided. Result: candidate lost.
I1014 19:27:26.746567 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:26.746623 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Leader election lost for term 598. Reason: could not achieve majority
I1014 19:27:26.754899 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 597. Current term is 598. Ops: []
I1014 19:27:26.755488 28683 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 597.444, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I1014 19:27:26.755719 28683 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 LEADER]: Stepping down as leader of term 597
I1014 19:27:26.755772 28683 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:26.755966 28683 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 597.444, Last appended by leader: 444, Current term: 597, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.756201 28683 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:26.756294 28683 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:26.756479 28683 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 597 FOLLOWER]: Advancing to term 598
I1014 19:27:26.831493 29154 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:26.831607 29154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:26.831682 29154 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 597 FOLLOWER]: Advancing to term 598
I1014 19:27:26.833041 29154 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.833406 29154 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 598 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.833585 29154 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 598 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:26.833967 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 598 candidate_status { last_received { term: 597 index: 444 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.834100 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:26.834144 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 598 candidate_status { last_received { term: 597 index: 444 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:26.834277 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 598: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.834596 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 598 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 598: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:26.835467 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 598.
I1014 19:27:26.835752 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 598 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.835811 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 598 election: Election decided. Result: candidate won.
I1014 19:27:26.835992 29154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:26.836091 29154 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 FOLLOWER]: Leader election won for term 598
I1014 19:27:26.836225 29154 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:26.836431 29154 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 444, Committed index: 444, Last appended: 597.444, Last appended by leader: 444, Current term: 598, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.843469 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.843689 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.843849 29156 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:26.845161 29156 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:26.846084 29156 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:26.846724 29156 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:26.848471 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 597 index: 444. Preceding OpId from leader: term: 598 index: 446. (index mismatch)
I1014 19:27:26.848541 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 597 index: 444. Preceding OpId from leader: term: 598 index: 446. (index mismatch)
I1014 19:27:26.848875 29154 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I1014 19:27:26.849059 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 445, Last known committed idx: 444, Time since last communication: 0.000s
I1014 19:27:26.850385 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.850608 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.850648 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.850853 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.852020 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.852188 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.852188 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.852357 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.851961 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.852608 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:26.853299 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 598 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:26.853530 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:26.953214 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:26.953296 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:26.953349 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 598 FOLLOWER]: Advancing to term 599
I1014 19:27:26.954980 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 599 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.955476 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 599 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:26.955618 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 599 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:26.956111 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 599 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:26.956182 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 599 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:26.956302 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 599 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.956367 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 599 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.956698 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 599 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 599 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.956849 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 599 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 599 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:26.956904 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 599 election: Election decided. Result: candidate lost.
I1014 19:27:26.957159 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:26.957216 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 599 FOLLOWER]: Leader election lost for term 599. Reason: could not achieve majority
I1014 19:27:26.973191 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 599 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 598. Current term is 599. Ops: []
I1014 19:27:26.973881 29427 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 598.446, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I1014 19:27:26.974148 29427 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 LEADER]: Stepping down as leader of term 598
I1014 19:27:26.974195 29427 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:26.974381 29427 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 598.446, Last appended by leader: 446, Current term: 598, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:26.974591 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:26.974638 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:26.974771 29427 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 598 FOLLOWER]: Advancing to term 599
I1014 19:27:27.031168 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.031293 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:27.031396 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 598 FOLLOWER]: Advancing to term 599
I1014 19:27:27.034085 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 599 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.034704 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 599 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.050863 29154 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 599 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:27.050946 29154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:27.050999 29154 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 599 FOLLOWER]: Advancing to term 600
I1014 19:27:27.052974 29154 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.053436 29154 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 600 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.053603 29154 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 600 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.054167 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 600 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.054319 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 599 FOLLOWER]: Advancing to term 600
I1014 19:27:27.054306 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 599 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.054390 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.140s (vote granted)
I1014 19:27:27.054447 28826 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 599. Current term is 600.
W1014 19:27:27.054826 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 599 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 599. Current term is 600.
I1014 19:27:27.054857 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 599 election: Cancelling election due to peer responding with higher term
I1014 19:27:27.056357 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 600 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 600.
I1014 19:27:27.056516 28683 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:27.056573 28683 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 600 FOLLOWER]: Leader election lost for term 599. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 599. Current term is 600.
I1014 19:27:27.056676 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 600 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.056835 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 599 FOLLOWER]: Advancing to term 600
I1014 19:27:27.056911 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.167s (vote granted)
I1014 19:27:27.058917 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 600.
I1014 19:27:27.059228 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 599 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.059278 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 600 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.059336 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 600 election: Election decided. Result: candidate won.
I1014 19:27:27.059571 29154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:27.059631 29154 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Leader election won for term 600
I1014 19:27:27.059674 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 600 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.059717 29154 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:27.059918 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 599 candidate_status { last_received { term: 598 index: 446 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.060068 29011 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 599. Current term is 600.
I1014 19:27:27.059967 29154 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 598.446, Last appended by leader: 446, Current term: 600, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
W1014 19:27:27.060451 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 599 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 599. Current term is 600.
I1014 19:27:27.060875 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.061066 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.061296 29156 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:27.063057 29156 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:27.064340 29156 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:27.065174 29156 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:27.067394 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 598 index: 446. Preceding OpId from leader: term: 600 index: 448. (index mismatch)
I1014 19:27:27.067585 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 600 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 598 index: 446. Preceding OpId from leader: term: 600 index: 448. (index mismatch)
I1014 19:27:27.067961 29427 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I1014 19:27:27.068213 29154 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I1014 19:27:27.070008 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.070289 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.070219 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.070405 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.072818 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.073002 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.072875 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.073117 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.073477 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.073658 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.074301 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 600 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.074506 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.174427 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 600 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.174525 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.149s (starting election)
I1014 19:27:27.174587 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 600 FOLLOWER]: Advancing to term 601
I1014 19:27:27.176715 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.177258 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 601 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.177456 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 601 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.178225 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 601 candidate_status { last_received { term: 600 index: 448 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.178416 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 601 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.178741 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 601 candidate_status { last_received { term: 600 index: 448 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.178952 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 601 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.179491 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 601 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 601 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.180210 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 601 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 601 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.180275 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 601 election: Election decided. Result: candidate lost.
I1014 19:27:27.180486 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:27.180552 28971 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Leader election lost for term 601. Reason: could not achieve majority
I1014 19:27:27.192001 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 600. Current term is 601. Ops: []
I1014 19:27:27.192479 28319 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 600.448, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I1014 19:27:27.192704 28319 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 LEADER]: Stepping down as leader of term 600
I1014 19:27:27.192739 28319 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:27.192889 28319 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 448, Committed index: 448, Last appended: 600.448, Last appended by leader: 448, Current term: 600, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.193049 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:27.193078 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:27.193178 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 600 FOLLOWER]: Advancing to term 601
I1014 19:27:27.254897 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.254983 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:27.255044 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 600 FOLLOWER]: Advancing to term 601
I1014 19:27:27.257223 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.257786 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 601 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.257952 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 601 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.258751 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 601 candidate_status { last_received { term: 600 index: 448 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.258918 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (vote granted)
I1014 19:27:27.258865 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 601 candidate_status { last_received { term: 600 index: 448 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.259045 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 601: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:27.259469 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 601 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 601: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:27.260881 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 601.
I1014 19:27:27.261234 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 601 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.261292 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 601 election: Election decided. Result: candidate won.
I1014 19:27:27.261548 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:27.261670 28528 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 FOLLOWER]: Leader election won for term 601
I1014 19:27:27.271013 28528 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:27.271361 28528 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 448, Committed index: 448, Last appended: 600.448, Last appended by leader: 448, Current term: 601, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.273610 29575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.273937 29575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.274541 29576 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:27.276473 29576 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:27.277773 29576 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:27.278645 29576 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:27.281147 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 600 index: 448. Preceding OpId from leader: term: 601 index: 450. (index mismatch)
I1014 19:27:27.281147 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 600 index: 448. Preceding OpId from leader: term: 601 index: 450. (index mismatch)
I1014 19:27:27.281664 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I1014 19:27:27.282702 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 449, Last known committed idx: 448, Time since last communication: 0.000s
I1014 19:27:27.285217 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.285437 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.285591 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.285756 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.287842 25847 mvcc.cc:203] Tried to move safe_time back from 6306080552057245696 to 6306080552027582464. Current Snapshot: MvccSnapshot[committed={T|T < 6306080552057245696}]
I1014 19:27:27.300062 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.300267 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.300400 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.300683 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.301725 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.301913 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.298804 29575 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 601 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.302655 29575 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.468044 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:27.468119 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:27:27.468169 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 601 FOLLOWER]: Advancing to term 602
I1014 19:27:27.470281 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.470793 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 602 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.470955 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 602 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.471401 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 602 candidate_status { last_received { term: 601 index: 450 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.471616 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 602 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.471616 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 602 candidate_status { last_received { term: 601 index: 450 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.471804 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 602 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.472134 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 602 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 602 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.472345 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 602 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 602 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.472398 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 602 election: Election decided. Result: candidate lost.
I1014 19:27:27.472580 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:27.472669 28971 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Leader election lost for term 602. Reason: could not achieve majority
I1014 19:27:27.476184 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 601. Current term is 602. Ops: []
I1014 19:27:27.476668 28528 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 601.450, Next index: 451, Last known committed idx: 450, Time since last communication: 0.000s
I1014 19:27:27.476915 28528 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 LEADER]: Stepping down as leader of term 601
I1014 19:27:27.476956 28528 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:27.477115 28528 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 450, Committed index: 450, Last appended: 601.450, Last appended by leader: 450, Current term: 601, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.477288 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:27.477319 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:27.477458 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 601 FOLLOWER]: Advancing to term 602
I1014 19:27:27.494815 29427 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:27.494887 29427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:27.494936 29427 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 601 FOLLOWER]: Advancing to term 602
I1014 19:27:27.496984 29427 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.497439 29427 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 602 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.497604 29427 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 602 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.498046 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 602 candidate_status { last_received { term: 601 index: 450 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.498237 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 602: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:27.498399 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 602 candidate_status { last_received { term: 601 index: 450 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.498574 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:27.498625 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 602 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 602: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:27.500550 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 602 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 602.
I1014 19:27:27.500879 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 602 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.500927 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 602 election: Election decided. Result: candidate won.
I1014 19:27:27.501227 29427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:27.501286 29427 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 FOLLOWER]: Leader election won for term 602
I1014 19:27:27.501657 29427 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:27.501826 29427 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 450, Committed index: 450, Last appended: 601.450, Last appended by leader: 450, Current term: 602, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.502848 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.502975 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.503132 29156 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:27.504472 29156 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:27.505439 29156 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:27.506084 29156 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:27.508437 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 602 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 601 index: 450. Preceding OpId from leader: term: 602 index: 452. (index mismatch)
I1014 19:27:27.508525 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 601 index: 450. Preceding OpId from leader: term: 602 index: 452. (index mismatch)
I1014 19:27:27.508922 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 451, Last known committed idx: 450, Time since last communication: 0.000s
I1014 19:27:27.509101 29427 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 451, Last known committed idx: 450, Time since last communication: 0.000s
I1014 19:27:27.510630 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.510820 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.510954 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.511140 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.512138 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.512320 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.512545 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.512706 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.513504 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.513702 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.513861 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 602 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.514017 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.613056 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 602 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.613129 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:27.613176 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 602 FOLLOWER]: Advancing to term 603
I1014 19:27:27.615214 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 603 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.615691 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 603 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.615839 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 603 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.616317 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 603 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.616369 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 603 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.616514 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 603 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.616539 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 603 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.616900 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 603 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 603 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.617427 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 603 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 602. Current term is 603. Ops: []
I1014 19:27:27.617722 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 603 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 603 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.617786 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 603 election: Election decided. Result: candidate lost.
I1014 19:27:27.618242 29427 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 602.452, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I1014 19:27:27.618444 28319 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 LEADER]: Stepping down as leader of term 602
I1014 19:27:27.618485 28319 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:27.618501 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:27.618552 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 603 FOLLOWER]: Leader election lost for term 603. Reason: could not achieve majority
I1014 19:27:27.618669 28319 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 452, Committed index: 452, Last appended: 602.452, Last appended by leader: 452, Current term: 602, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.618834 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:27.618872 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:27.618988 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 602 FOLLOWER]: Advancing to term 603
I1014 19:27:27.714104 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.714185 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:27.714210 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 603 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:27.714242 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 602 FOLLOWER]: Advancing to term 603
I1014 19:27:27.714277 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:27.714320 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 603 FOLLOWER]: Advancing to term 604
I1014 19:27:27.715801 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.716187 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 604 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.716337 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 604 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.716333 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 603 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.716812 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 603 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.716866 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 604 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.716955 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 603 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.717005 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 603 FOLLOWER]: Advancing to term 604
I1014 19:27:27.716980 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 604 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.717092 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (vote granted)
I1014 19:27:27.717142 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 603 FOLLOWER]: Advancing to term 604
I1014 19:27:27.717214 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:27.719010 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 604.
I1014 19:27:27.719199 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 604.
I1014 19:27:27.719682 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 604 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.719738 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 604 election: Election decided. Result: candidate won.
I1014 19:27:27.719724 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 603 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:27.719880 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 603. Current term is 604.
I1014 19:27:27.720000 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 603 candidate_status { last_received { term: 602 index: 452 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.720160 28826 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 603. Current term is 604.
I1014 19:27:27.720207 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:27.720245 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 604 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:27.720259 28319 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 FOLLOWER]: Leader election won for term 604
I1014 19:27:27.720329 28319 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
W1014 19:27:27.720559 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 603 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 603. Current term is 604.
I1014 19:27:27.720585 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 603 election: Cancelling election due to peer responding with higher term
I1014 19:27:27.720526 28319 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 452, Committed index: 452, Last appended: 602.452, Last appended by leader: 452, Current term: 604, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
W1014 19:27:27.720914 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 603 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 603. Current term is 604.
I1014 19:27:27.720902 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:27.720978 28971 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Leader election lost for term 603. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 603. Current term is 604.
I1014 19:27:27.721451 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.721653 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.722139 29156 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:27.724016 29156 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:27.725131 29156 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:27.726001 29156 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:27.728197 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 602 index: 452. Preceding OpId from leader: term: 604 index: 454. (index mismatch)
I1014 19:27:27.728219 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 602 index: 452. Preceding OpId from leader: term: 604 index: 454. (index mismatch)
I1014 19:27:27.728754 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I1014 19:27:27.729190 28319 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I1014 19:27:27.730397 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.730619 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.732358 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.732564 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.733186 29154 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.733366 29154 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.733747 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.733932 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:27.734149 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.734331 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.734763 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 604 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:27.734938 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:27.735107 25850 mvcc.cc:203] Tried to move safe_time back from 6306080553889181696 to 6306080553865904128. Current Snapshot: MvccSnapshot[committed={T|T < 6306080553889181696 or (T in {6306080553889181696})}]
I1014 19:27:27.915333 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:27.915427 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (starting election)
I1014 19:27:27.915504 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 604 FOLLOWER]: Advancing to term 605
I1014 19:27:27.917603 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.918087 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 605 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:27.918247 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 605 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:27.919066 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 605 candidate_status { last_received { term: 604 index: 454 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:27.919066 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 605 candidate_status { last_received { term: 604 index: 454 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:27.919247 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 605 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.919252 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 605 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.919641 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 605 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 605 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.919781 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 605 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 605 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:27.919838 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 605 election: Election decided. Result: candidate lost.
I1014 19:27:27.920043 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:27.920095 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Leader election lost for term 605. Reason: could not achieve majority
I1014 19:27:27.927654 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 604. Current term is 605. Ops: []
I1014 19:27:27.928184 29427 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 604.454, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I1014 19:27:27.928416 28319 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 LEADER]: Stepping down as leader of term 604
I1014 19:27:27.928486 28319 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:27.928663 28319 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 454, Committed index: 454, Last appended: 604.454, Last appended by leader: 454, Current term: 604, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:27.928835 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:27.928865 28319 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:27.928967 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 604 FOLLOWER]: Advancing to term 605
I1014 19:27:28.000464 28971 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:28.000541 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:28.000588 28971 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 604 FOLLOWER]: Advancing to term 605
I1014 19:27:28.002637 28971 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.003099 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 605 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.003238 28971 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 605 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.003870 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 605 candidate_status { last_received { term: 604 index: 454 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.004104 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 605: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.004456 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 605 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 605: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.004449 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 605 candidate_status { last_received { term: 604 index: 454 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.004606 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (vote granted)
I1014 19:27:28.006693 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 605.
I1014 19:27:28.007102 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 605 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.007165 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 605 election: Election decided. Result: candidate won.
I1014 19:27:28.007359 28971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:28.007428 28971 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 FOLLOWER]: Leader election won for term 605
I1014 19:27:28.007750 28971 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:28.007952 28971 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 454, Committed index: 454, Last appended: 604.454, Last appended by leader: 454, Current term: 605, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.008891 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.011030 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.015972 29818 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.017949 29818 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.019220 29818 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.020071 29818 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:28.022485 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 604 index: 454. Preceding OpId from leader: term: 605 index: 456. (index mismatch)
I1014 19:27:28.023008 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 604 index: 454. Preceding OpId from leader: term: 605 index: 456. (index mismatch)
I1014 19:27:28.023494 28971 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I1014 19:27:28.024839 28971 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I1014 19:27:28.025810 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.026002 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.027410 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.027616 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.028394 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.028609 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.029824 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.029981 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.030875 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.031111 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.032528 28319 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 605 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.032708 28319 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.131350 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:28.131441 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:27:28.131526 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 605 FOLLOWER]: Advancing to term 606
I1014 19:27:28.133792 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.134320 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 606 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.134500 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 606 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.135008 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 606 candidate_status { last_received { term: 605 index: 456 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.135051 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 606 candidate_status { last_received { term: 605 index: 456 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.135192 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 606 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.135222 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 606 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.135677 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 606 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 606 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.136169 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 606 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 606 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.136232 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 606 election: Election decided. Result: candidate lost.
I1014 19:27:28.136608 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:28.136703 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Leader election lost for term 606. Reason: could not achieve majority
I1014 19:27:28.150236 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 605. Current term is 606. Ops: []
I1014 19:27:28.151008 28971 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 605.456, Next index: 457, Last known committed idx: 456, Time since last communication: 0.000s
I1014 19:27:28.151237 29817 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 LEADER]: Stepping down as leader of term 605
I1014 19:27:28.151278 29817 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:28.151463 29817 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 456, Committed index: 456, Last appended: 605.456, Last appended by leader: 456, Current term: 605, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.151607 29817 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:28.151644 29817 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:28.151757 29817 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 605 FOLLOWER]: Advancing to term 606
I1014 19:27:28.227785 28319 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:28.227864 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:28.227915 28319 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 605 FOLLOWER]: Advancing to term 606
I1014 19:27:28.229785 28319 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.230254 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 606 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.230443 28319 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 606 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.231122 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 606 candidate_status { last_received { term: 605 index: 456 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.231284 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:28.233284 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 606.
I1014 19:27:28.234766 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 606 candidate_status { last_received { term: 605 index: 456 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.234944 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 606: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.235801 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 606 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 606: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.236186 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 606 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.236240 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 606 election: Election decided. Result: candidate won.
I1014 19:27:28.237068 28319 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:28.237125 28319 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 FOLLOWER]: Leader election won for term 606
I1014 19:27:28.237313 28319 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:28.237521 28319 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 456, Committed index: 456, Last appended: 605.456, Last appended by leader: 456, Current term: 606, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.238402 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.238584 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.239066 29891 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.241050 29891 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.242532 29891 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.243438 29891 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:28.245802 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 605 index: 456. Preceding OpId from leader: term: 606 index: 458. (index mismatch)
I1014 19:27:28.246042 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 605 index: 456. Preceding OpId from leader: term: 606 index: 458. (index mismatch)
I1014 19:27:28.246575 29427 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 457, Last known committed idx: 456, Time since last communication: 0.000s
I1014 19:27:28.246909 29427 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 457, Last known committed idx: 456, Time since last communication: 0.000s
I1014 19:27:28.248154 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.248351 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.248688 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.248867 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.252213 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.252467 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.252627 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.252813 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.253007 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.253190 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.253406 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 606 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.253605 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.353066 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:28.353152 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:28.353215 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 606 FOLLOWER]: Advancing to term 607
I1014 19:27:28.354703 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.355123 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 607 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.355273 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 607 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.356169 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 607 candidate_status { last_received { term: 606 index: 458 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.356169 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 607 candidate_status { last_received { term: 606 index: 458 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.356356 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 607 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.356355 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 607 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.356875 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 607 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 607 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.357026 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 607 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 607 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.357084 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 607 election: Election decided. Result: candidate lost.
I1014 19:27:28.357259 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:28.357311 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Leader election lost for term 607. Reason: could not achieve majority
I1014 19:27:28.374476 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 606. Current term is 607. Ops: []
I1014 19:27:28.375031 29893 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 606.458, Next index: 459, Last known committed idx: 458, Time since last communication: 0.000s
I1014 19:27:28.375304 29893 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 LEADER]: Stepping down as leader of term 606
I1014 19:27:28.375340 29893 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:28.375532 29893 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 458, Committed index: 458, Last appended: 606.458, Last appended by leader: 458, Current term: 606, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.375715 29893 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:28.375747 29893 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:28.375854 29893 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 606 FOLLOWER]: Advancing to term 607
I1014 19:27:28.437211 29817 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:28.437283 29817 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:28.437340 29817 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 606 FOLLOWER]: Advancing to term 607
I1014 19:27:28.439479 29817 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.439954 29817 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 607 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.440099 29817 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 607 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.440881 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 607 candidate_status { last_received { term: 606 index: 458 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.440881 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 607 candidate_status { last_received { term: 606 index: 458 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.441082 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 607: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.441053 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (vote granted)
I1014 19:27:28.442397 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 607 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 607: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:28.443099 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 607 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 607.
I1014 19:27:28.443418 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 607 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.443495 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 607 election: Election decided. Result: candidate won.
I1014 19:27:28.443688 29817 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:28.443744 29817 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 FOLLOWER]: Leader election won for term 607
I1014 19:27:28.443828 29817 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:28.444022 29817 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 458, Committed index: 458, Last appended: 606.458, Last appended by leader: 458, Current term: 607, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.444983 28971 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.445169 28971 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.445338 29818 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.446952 29818 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.448259 29818 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.449172 29818 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:28.452255 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 606 index: 458. Preceding OpId from leader: term: 607 index: 460. (index mismatch)
I1014 19:27:28.452702 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 607 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 606 index: 458. Preceding OpId from leader: term: 607 index: 460. (index mismatch)
I1014 19:27:28.453362 28971 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 459, Last known committed idx: 458, Time since last communication: 0.000s
I1014 19:27:28.453572 29817 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 459, Last known committed idx: 458, Time since last communication: 0.000s
I1014 19:27:28.455217 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.455468 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.455660 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.455859 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.457119 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.457296 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.466825 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.467036 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.467514 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.467682 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.468435 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 607 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.468648 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.568801 29893 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 607 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:28.568882 29893 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:28.568936 29893 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 607 FOLLOWER]: Advancing to term 608
I1014 19:27:28.571305 29893 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.571861 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 608 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.572005 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 608 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.572672 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 608 candidate_status { last_received { term: 607 index: 460 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.572865 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 608 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.572901 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 608 candidate_status { last_received { term: 607 index: 460 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.573071 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 608 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.573245 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 608 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 608 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.573523 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 608 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 608 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.573593 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 608 election: Election decided. Result: candidate lost.
I1014 19:27:28.573799 29427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:28.573859 29427 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Leader election lost for term 608. Reason: could not achieve majority
I1014 19:27:28.591801 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 607. Current term is 608. Ops: []
I1014 19:27:28.592427 29817 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 607.460, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I1014 19:27:28.592694 29945 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 LEADER]: Stepping down as leader of term 607
I1014 19:27:28.592738 29945 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:28.592980 29945 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 460, Committed index: 460, Last appended: 607.460, Last appended by leader: 460, Current term: 607, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.593183 29945 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:28.593224 29945 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:28.593372 29945 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 607 FOLLOWER]: Advancing to term 608
I1014 19:27:28.649852 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:28.649933 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:28.649981 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 607 FOLLOWER]: Advancing to term 608
I1014 19:27:28.651582 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.652098 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 608 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.652268 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 608 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.652881 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 608 candidate_status { last_received { term: 607 index: 460 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.652910 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 608 candidate_status { last_received { term: 607 index: 460 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.653043 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (vote granted)
I1014 19:27:28.653107 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 608: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:28.653599 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 608 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 608: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:28.655056 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 608 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 608.
I1014 19:27:28.655508 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 608 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.655568 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 608 election: Election decided. Result: candidate won.
I1014 19:27:28.655789 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:28.655853 28528 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 FOLLOWER]: Leader election won for term 608
I1014 19:27:28.656237 28528 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:28.656527 28528 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 460, Committed index: 460, Last appended: 607.460, Last appended by leader: 460, Current term: 608, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.657954 30033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.658211 30033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.664685 30034 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.667090 30034 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.668345 30034 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.669363 30034 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:28.671437 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 607 index: 460. Preceding OpId from leader: term: 608 index: 462. (index mismatch)
I1014 19:27:28.671458 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 608 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 607 index: 460. Preceding OpId from leader: term: 608 index: 462. (index mismatch)
I1014 19:27:28.671963 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I1014 19:27:28.672807 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 461, Last known committed idx: 460, Time since last communication: 0.000s
I1014 19:27:28.674003 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.674242 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.674463 29945 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.674649 29945 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.675283 30033 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.675467 30033 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.676127 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.676295 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.676926 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.677186 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.678050 29945 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 608 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.678205 29945 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:28.858072 29945 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 608 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:28.858147 29945 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:28.858201 29945 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 608 FOLLOWER]: Advancing to term 609
I1014 19:27:28.860297 29945 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.860764 29945 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 609 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.860939 29945 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 609 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.861502 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 609 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.861507 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 609 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.861690 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 609 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.861691 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 609 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.862080 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 609 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 609 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.862174 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 609 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 609 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.862238 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 609 election: Election decided. Result: candidate lost.
I1014 19:27:28.862500 29945 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:28.862552 29945 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Leader election lost for term 609. Reason: could not achieve majority
I1014 19:27:28.872053 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 608. Current term is 609. Ops: []
I1014 19:27:28.872539 30036 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 608.462, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I1014 19:27:28.872818 28528 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 LEADER]: Stepping down as leader of term 608
I1014 19:27:28.872861 28528 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:28.873025 28528 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 608.462, Last appended by leader: 462, Current term: 608, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.873203 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:28.873242 28528 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:28.873353 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 608 FOLLOWER]: Advancing to term 609
I1014 19:27:28.967255 29893 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:28.967322 29893 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:28.967381 29893 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 608 FOLLOWER]: Advancing to term 609
I1014 19:27:28.969019 29893 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.969385 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 609 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:28.969547 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 609 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.969938 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 609 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:28.970132 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 609: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:28.970057 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 609 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.970250 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (vote granted)
I1014 19:27:28.970438 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 609 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 609: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:28.971623 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 609 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 609.
I1014 19:27:28.971886 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 609 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.971933 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 609 election: Election decided. Result: candidate won.
I1014 19:27:28.972064 29893 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:28.972115 29893 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 FOLLOWER]: Leader election won for term 609
I1014 19:27:28.972198 29893 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:28.972398 29893 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 608.462, Last appended by leader: 462, Current term: 609, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.973122 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 609 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.973248 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.973659 30164 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.975119 30164 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.976130 29945 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:28.976203 29945 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.155s (starting election)
I1014 19:27:28.976249 29945 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 609 FOLLOWER]: Advancing to term 610
I1014 19:27:28.976825 30164 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.978200 29945 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.978482 30164 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:28.978664 29945 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 610 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:28.978809 29945 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 610 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.979235 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 610 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:28.979400 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 610 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.979420 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 610 candidate_status { last_received { term: 608 index: 462 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:28.979598 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 609 FOLLOWER]: Advancing to term 610
I1014 19:27:28.979661 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:28.980862 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 609. Current term is 610. Ops: []
I1014 19:27:28.981340 29893 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I1014 19:27:28.981544 29427 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 LEADER]: Stepping down as leader of term 609
I1014 19:27:28.981555 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 610 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 610.
I1014 19:27:28.981586 29427 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:28.981891 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 610 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:28.981751 29427 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 609.464, Last appended by leader: 464, Current term: 609, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.981959 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:28.981951 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 610 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 610 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:28.981997 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:28.982028 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 610 election: Election decided. Result: candidate won.
I1014 19:27:28.982084 29427 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 609 FOLLOWER]: Advancing to term 610
I1014 19:27:28.982309 29945 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.194s (election complete)
I1014 19:27:28.982365 29945 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 FOLLOWER]: Leader election won for term 610
I1014 19:27:28.982465 29945 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:28.982627 29945 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 462, Committed index: 462, Last appended: 608.462, Last appended by leader: 462, Current term: 610, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:28.983656 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 610 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 609. Current term is 610. Ops: []
I1014 19:27:28.983832 29817 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:28.984010 29817 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:28.990195 30165 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:28.991673 30165 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:28.992646 30165 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:28.993382 30165 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:28.998564 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 609 index: 464. Preceding OpId from leader: term: 610 index: 464. (term mismatch)
I1014 19:27:28.998569 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 610 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 608 index: 462. Preceding OpId from leader: term: 610 index: 464. (index mismatch)
I1014 19:27:28.998623 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 463
I1014 19:27:28.998646 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted WRITE_OP operation due to leader change: 609.464
I1014 19:27:28.998992 30164 catalog_manager.cc:1011] Loading token signing keys failed; change in term detected: 609 vs 610: Aborted: Transaction aborted by new leader
I1014 19:27:28.999337 29945 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I1014 19:27:28.999541 29817 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 463, Last known committed idx: 462, Time since last communication: 0.000s
I1014 19:27:28.999686 29817 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 609.463 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 462
I1014 19:27:29.001533 28826 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 462
I1014 19:27:29.001561 28826 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 609.463
I1014 19:27:29.001600 28826 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:29.001724 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.001902 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.002370 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.002584 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.002918 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.003155 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.011871 29945 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.012068 29945 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.012235 29945 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.012400 29945 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.014303 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 610 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.014523 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.103173 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 610 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:29.103260 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:29.103308 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 610 FOLLOWER]: Advancing to term 611
I1014 19:27:29.105609 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.106114 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 611 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.106271 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 611 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.106887 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 611 candidate_status { last_received { term: 610 index: 464 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:29.107079 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 611 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.107085 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 611 candidate_status { last_received { term: 610 index: 464 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:29.107260 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 611 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.107532 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 611 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 611 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.107699 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 611 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 611 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.107758 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 611 election: Election decided. Result: candidate lost.
I1014 19:27:29.107952 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (election complete)
I1014 19:27:29.108012 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Leader election lost for term 611. Reason: could not achieve majority
I1014 19:27:29.122056 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 610. Current term is 611. Ops: []
I1014 19:27:29.122664 30167 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 610.464, Next index: 465, Last known committed idx: 464, Time since last communication: 0.000s
I1014 19:27:29.122910 30167 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 LEADER]: Stepping down as leader of term 610
I1014 19:27:29.122943 30167 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:29.123092 30167 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 464, Committed index: 464, Last appended: 610.464, Last appended by leader: 464, Current term: 610, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.123256 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:29.123286 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:29.123386 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 610 FOLLOWER]: Advancing to term 611
I1014 19:27:29.200563 29427 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:29.200644 29427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:29.200692 29427 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 610 FOLLOWER]: Advancing to term 611
I1014 19:27:29.202462 29427 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.202924 29427 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 611 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.203078 29427 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 611 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:29.203564 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 611 candidate_status { last_received { term: 610 index: 464 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:29.203727 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:29.203936 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 611 candidate_status { last_received { term: 610 index: 464 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:29.204134 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 611: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:29.204505 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 611 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 611: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:29.205745 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 611 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 611.
I1014 19:27:29.206034 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 611 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.206094 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 611 election: Election decided. Result: candidate won.
I1014 19:27:29.206274 29427 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:29.206326 29427 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 FOLLOWER]: Leader election won for term 611
I1014 19:27:29.206400 29427 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:29.206594 29427 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 464, Committed index: 464, Last appended: 610.464, Last appended by leader: 464, Current term: 611, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.207787 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.207989 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.208153 30164 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:29.209903 30164 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:29.211342 30164 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:29.212188 30164 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:29.214097 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 610 index: 464. Preceding OpId from leader: term: 611 index: 466. (index mismatch)
I1014 19:27:29.214424 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 611 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 610 index: 464. Preceding OpId from leader: term: 611 index: 466. (index mismatch)
I1014 19:27:29.214582 29893 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 465, Last known committed idx: 464, Time since last communication: 0.000s
I1014 19:27:29.215010 29893 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 465, Last known committed idx: 464, Time since last communication: 0.000s
I1014 19:27:29.216215 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.216485 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.217452 29893 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.217635 29893 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.218513 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.218770 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.219331 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.219547 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.220106 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.220325 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.221285 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 611 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.221460 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.319010 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:29.319095 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:29.319155 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 611 FOLLOWER]: Advancing to term 612
I1014 19:27:29.321595 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.322124 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 612 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.322319 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 612 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.322758 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 611 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:29.322824 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:29.322871 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 611 FOLLOWER]: Advancing to term 612
I1014 19:27:29.322852 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 612 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:29.324893 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 612 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.325202 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 612 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 612: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:29.325347 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 612 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.325484 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 612 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:29.325675 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 612 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 612 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 612: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:29.325773 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 612 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:29.325978 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 612 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.326311 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 612 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 612 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.326375 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 612 election: Election decided. Result: candidate lost.
I1014 19:27:29.326367 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 612 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:29.326548 29011 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 612: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:29.326616 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:29.326668 28528 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Leader election lost for term 612. Reason: could not achieve majority
I1014 19:27:29.326849 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 612 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 612: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:29.326910 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 612 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:29.327059 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 612 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.327324 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 612 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 612 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.327385 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 612 election: Election decided. Result: candidate lost.
I1014 19:27:29.327673 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:29.327718 30167 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 612 FOLLOWER]: Leader election lost for term 612. Reason: could not achieve majority
I1014 19:27:29.330233 29011 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 611. Current term is 612. Ops: []
I1014 19:27:29.330649 29427 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 611.466, Next index: 467, Last known committed idx: 466, Time since last communication: 0.000s
I1014 19:27:29.330852 29893 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Stepping down as leader of term 611
I1014 19:27:29.330891 29893 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:29.331070 29893 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 466, Committed index: 466, Last appended: 611.466, Last appended by leader: 466, Current term: 611, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.331269 29893 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:29.331310 29893 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:29.331460 29893 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 611 FOLLOWER]: Advancing to term 612
I1014 19:27:29.438217 29893 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 612 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:29.438287 29893 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:29.438341 29893 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 612 FOLLOWER]: Advancing to term 613
I1014 19:27:29.439802 29893 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.440171 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 613 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.440282 29893 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 613 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:29.440757 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 613 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:29.440820 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 613 candidate_status { last_received { term: 611 index: 466 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:29.440909 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 612 FOLLOWER]: Advancing to term 613
I1014 19:27:29.440974 29011 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 612 FOLLOWER]: Advancing to term 613
I1014 19:27:29.440977 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.201s (vote granted)
I1014 19:27:29.441066 29011 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (vote granted)
I1014 19:27:29.443050 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 613 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 613.
I1014 19:27:29.443099 29011 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 613.
I1014 19:27:29.443347 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 613 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.443456 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 613 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:29.443502 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 613 election: Election decided. Result: candidate won.
I1014 19:27:29.443694 29893 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:29.443747 29893 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 FOLLOWER]: Leader election won for term 613
I1014 19:27:29.443828 29893 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:29.443982 29893 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 466, Committed index: 466, Last appended: 611.466, Last appended by leader: 466, Current term: 613, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.444793 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.444916 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.445077 30164 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:29.446722 30164 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:29.447711 30164 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:29.448491 30164 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:29.450296 29011 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 611 index: 466. Preceding OpId from leader: term: 613 index: 468. (index mismatch)
I1014 19:27:29.450359 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 613 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 611 index: 466. Preceding OpId from leader: term: 613 index: 468. (index mismatch)
I1014 19:27:29.450747 29893 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 467, Last known committed idx: 466, Time since last communication: 0.000s
I1014 19:27:29.450928 29427 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 467, Last known committed idx: 466, Time since last communication: 0.000s
I1014 19:27:29.452461 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.452392 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.452646 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.452703 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.454479 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.454700 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.455996 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.456193 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.456356 29427 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.456598 29427 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.456146 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 613 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.456707 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.830848 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 613 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:29.830920 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:29.830970 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 613 FOLLOWER]: Advancing to term 614
I1014 19:27:29.832885 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.833403 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 614 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.833598 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 614 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:29.834380 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 614 candidate_status { last_received { term: 613 index: 468 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:29.834596 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 614 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.834928 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 614 candidate_status { last_received { term: 613 index: 468 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:29.835122 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 614 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.835515 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 614 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 614 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.835821 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 614 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 614 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:29.835882 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 614 election: Election decided. Result: candidate lost.
I1014 19:27:29.836068 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:29.836114 30167 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Leader election lost for term 614. Reason: could not achieve majority
I1014 19:27:29.841249 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 613. Current term is 614. Ops: []
I1014 19:27:29.842772 29427 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 613.468, Next index: 469, Last known committed idx: 468, Time since last communication: 0.000s
I1014 19:27:29.843026 29427 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 LEADER]: Stepping down as leader of term 613
I1014 19:27:29.843061 29427 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:29.843227 29427 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 468, Committed index: 468, Last appended: 613.468, Last appended by leader: 468, Current term: 613, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.843441 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:29.843482 29427 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:29.843595 29427 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 613 FOLLOWER]: Advancing to term 614
I1014 19:27:29.893882 28528 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:29.893963 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:29.894017 28528 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 613 FOLLOWER]: Advancing to term 614
I1014 19:27:29.895682 28528 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.896185 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 614 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:29.896356 28528 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 614 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.896904 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 614 candidate_status { last_received { term: 613 index: 468 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:29.896919 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 614 candidate_status { last_received { term: 613 index: 468 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:29.897076 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 614: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:29.897063 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (vote granted)
I1014 19:27:29.897429 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 614 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 614: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:29.905625 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 614 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 614.
I1014 19:27:29.905951 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 614 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:29.906028 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 614 election: Election decided. Result: candidate won.
I1014 19:27:29.906327 28528 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:29.906409 28528 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 FOLLOWER]: Leader election won for term 614
I1014 19:27:29.906780 28528 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:29.906998 28528 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 468, Committed index: 468, Last appended: 613.468, Last appended by leader: 468, Current term: 614, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:29.908041 30556 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.908339 30556 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.908824 30557 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:29.910303 30557 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:29.911322 30557 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:29.911984 30557 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:29.913709 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 614 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 613 index: 468. Preceding OpId from leader: term: 614 index: 470. (index mismatch)
I1014 19:27:29.913745 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 613 index: 468. Preceding OpId from leader: term: 614 index: 470. (index mismatch)
I1014 19:27:29.914130 30556 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 469, Last known committed idx: 468, Time since last communication: 0.000s
I1014 19:27:29.914324 28528 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 469, Last known committed idx: 468, Time since last communication: 0.000s
I1014 19:27:29.915570 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.915741 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.915715 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.915900 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.917903 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.918129 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.918341 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.919361 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:29.919349 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.919644 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:29.917304 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 614 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:29.919934 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.017865 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 614 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.017951 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:30.018013 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 614 FOLLOWER]: Advancing to term 615
I1014 19:27:30.018229 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.018286 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:30.018334 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 614 FOLLOWER]: Advancing to term 615
I1014 19:27:30.019444 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.019829 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 615 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.019841 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 615 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.019968 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 615 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.020220 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 615 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.020351 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 615 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.020375 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 615 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.020565 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 615 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 615: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:30.020952 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 615 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 615 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 615: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:30.021046 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 615 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.021212 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 615: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.021682 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 615 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 615: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.021728 29011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 615 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.021956 29011 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 615 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.021850 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 615 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.022156 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 615 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.022498 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 615 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 615 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.022562 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 615 election: Election decided. Result: candidate lost.
I1014 19:27:30.023046 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 615 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 615 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.023094 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:30.023113 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 615 election: Election decided. Result: candidate lost.
I1014 19:27:30.023146 30167 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 615 FOLLOWER]: Leader election lost for term 615. Reason: could not achieve majority
I1014 19:27:30.023344 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:30.023393 30382 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Leader election lost for term 615. Reason: could not achieve majority
I1014 19:27:30.036303 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 614. Current term is 615. Ops: []
I1014 19:27:30.036890 30559 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 614.470, Next index: 471, Last known committed idx: 470, Time since last communication: 0.000s
I1014 19:27:30.037176 30556 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Stepping down as leader of term 614
I1014 19:27:30.037241 30556 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:30.037451 30556 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 470, Committed index: 470, Last appended: 614.470, Last appended by leader: 470, Current term: 614, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.037672 30556 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:30.037737 30556 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:30.037881 30556 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 614 FOLLOWER]: Advancing to term 615
I1014 19:27:30.138254 30556 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 615 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:30.138339 30556 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:30.138414 30556 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 615 FOLLOWER]: Advancing to term 616
I1014 19:27:30.139950 30556 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.140328 30556 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 616 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.140518 30556 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 616 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.140949 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 616 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.140992 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 616 candidate_status { last_received { term: 614 index: 470 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.141083 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 615 FOLLOWER]: Advancing to term 616
I1014 19:27:30.141134 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 615 FOLLOWER]: Advancing to term 616
I1014 19:27:30.141142 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.194s (vote granted)
I1014 19:27:30.141189 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.165s (vote granted)
I1014 19:27:30.142457 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 616.
I1014 19:27:30.142953 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 616 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.143055 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 616 election: Election decided. Result: candidate won.
I1014 19:27:30.143118 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 616 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 616.
I1014 19:27:30.143366 30556 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:30.143442 30556 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 FOLLOWER]: Leader election won for term 616
I1014 19:27:30.143632 30556 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:30.143812 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 616 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.143896 30556 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 470, Committed index: 470, Last appended: 614.470, Last appended by leader: 470, Current term: 616, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.145007 30559 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.145196 30559 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.145391 30557 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:30.147227 30557 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:30.148542 30557 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:30.149369 30557 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:30.151196 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 616 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 614 index: 470. Preceding OpId from leader: term: 616 index: 472. (index mismatch)
I1014 19:27:30.151253 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 614 index: 470. Preceding OpId from leader: term: 616 index: 472. (index mismatch)
I1014 19:27:30.151778 30556 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 471, Last known committed idx: 470, Time since last communication: 0.000s
I1014 19:27:30.151928 30559 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 471, Last known committed idx: 470, Time since last communication: 0.000s
I1014 19:27:30.153535 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.153719 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.153990 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.154182 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.155462 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.155670 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.155839 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.156113 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.156924 30556 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.157099 30556 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.157366 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 616 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.157550 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.342586 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 616 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.342677 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:30.342730 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 616 FOLLOWER]: Advancing to term 617
I1014 19:27:30.344779 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.345257 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 617 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.345425 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 617 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.345995 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 617 candidate_status { last_received { term: 616 index: 472 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.345996 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 617 candidate_status { last_received { term: 616 index: 472 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.346170 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 617 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.346184 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 617 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.346536 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 617 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 617 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.346612 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 617 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 617 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.346685 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 617 election: Election decided. Result: candidate lost.
I1014 19:27:30.346971 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:30.347030 30382 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Leader election lost for term 617. Reason: could not achieve majority
I1014 19:27:30.365134 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 616. Current term is 617. Ops: []
I1014 19:27:30.365772 30556 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 616.472, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I1014 19:27:30.365993 30558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 LEADER]: Stepping down as leader of term 616
I1014 19:27:30.366032 30558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:30.366243 30558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 472, Committed index: 472, Last appended: 616.472, Last appended by leader: 472, Current term: 616, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.366425 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:30.366463 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:30.366611 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 616 FOLLOWER]: Advancing to term 617
I1014 19:27:30.424252 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.424332 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:30.424386 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 616 FOLLOWER]: Advancing to term 617
I1014 19:27:30.426049 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.432670 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 617 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.432873 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 617 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.433311 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 617 candidate_status { last_received { term: 616 index: 472 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.433451 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 617: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.433389 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 617 candidate_status { last_received { term: 616 index: 472 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.433562 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (vote granted)
I1014 19:27:30.433784 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 617 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 617: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.434890 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 617.
I1014 19:27:30.435184 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 617 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.435230 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 617 election: Election decided. Result: candidate won.
I1014 19:27:30.435420 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:30.435479 30167 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 FOLLOWER]: Leader election won for term 617
I1014 19:27:30.435793 30167 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:30.435950 30167 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 472, Committed index: 472, Last appended: 616.472, Last appended by leader: 472, Current term: 617, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.436620 30733 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.436741 30733 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.437067 30734 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:30.438349 30734 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:30.439258 30734 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:30.439862 30734 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:30.441421 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 616 index: 472. Preceding OpId from leader: term: 617 index: 474. (index mismatch)
I1014 19:27:30.441421 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 616 index: 472. Preceding OpId from leader: term: 617 index: 474. (index mismatch)
I1014 19:27:30.441844 30167 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I1014 19:27:30.441983 30733 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 473, Last known committed idx: 472, Time since last communication: 0.000s
I1014 19:27:30.443650 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.443861 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.443733 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.443907 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.445502 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.445669 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.446106 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.446326 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.446486 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.446429 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 617 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.446686 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.446720 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.545732 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:30.545815 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:30.545873 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 617 FOLLOWER]: Advancing to term 618
I1014 19:27:30.547956 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.548401 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 618 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.548573 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 618 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.549064 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 618 candidate_status { last_received { term: 617 index: 474 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.549248 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 618 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.549342 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 618 candidate_status { last_received { term: 617 index: 474 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.549544 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 618 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.549625 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 618 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 618 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.550181 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 618 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 618 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.550240 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 618 election: Election decided. Result: candidate lost.
I1014 19:27:30.550426 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:30.550510 30382 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Leader election lost for term 618. Reason: could not achieve majority
I1014 19:27:30.550593 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 617. Current term is 618. Ops: []
I1014 19:27:30.551015 30735 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 617.474, Next index: 475, Last known committed idx: 474, Time since last communication: 0.000s
I1014 19:27:30.551280 30167 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 LEADER]: Stepping down as leader of term 617
I1014 19:27:30.551319 30167 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:30.551508 30167 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 474, Committed index: 474, Last appended: 617.474, Last appended by leader: 474, Current term: 617, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.551676 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:30.551709 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:30.551817 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 617 FOLLOWER]: Advancing to term 618
I1014 19:27:30.634215 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:30.634289 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:30.634337 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 617 FOLLOWER]: Advancing to term 618
I1014 19:27:30.635288 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
I1014 19:27:30.636353 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.636838 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 618 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.636967 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 618 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.637421 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 618 candidate_status { last_received { term: 617 index: 474 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.637435 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 618 candidate_status { last_received { term: 617 index: 474 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.637589 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (vote granted)
I1014 19:27:30.637614 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 618: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.637949 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 618 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 618: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:30.639225 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 618 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 618.
I1014 19:27:30.639582 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 618 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.639657 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 618 election: Election decided. Result: candidate won.
I1014 19:27:30.640000 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:30.640060 30558 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 FOLLOWER]: Leader election won for term 618
I1014 19:27:30.640148 30558 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:30.640311 30558 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 474, Committed index: 474, Last appended: 617.474, Last appended by leader: 474, Current term: 618, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.641409 30556 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.641619 30556 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.641978 30557 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:30.643649 30557 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:30.644605 30557 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:30.645227 30557 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:30.646898 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 618 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 617 index: 474. Preceding OpId from leader: term: 618 index: 476. (index mismatch)
I1014 19:27:30.647033 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 617 index: 474. Preceding OpId from leader: term: 618 index: 476. (index mismatch)
I1014 19:27:30.647343 30558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 475, Last known committed idx: 474, Time since last communication: 0.000s
I1014 19:27:30.647532 30556 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 475, Last known committed idx: 474, Time since last communication: 0.000s
I1014 19:27:30.648962 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.649142 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.649426 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.649682 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.650266 28528 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.650439 28528 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.651351 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.651549 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.651407 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.651588 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.650331 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 618 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.652153 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.843190 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 618 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.843279 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:30.843330 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 618 FOLLOWER]: Advancing to term 619
I1014 19:27:30.845340 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.845825 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 619 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.845983 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 619 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.846426 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 619 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.846621 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.846994 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 619 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.847311 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 619 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.847519 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.848420 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 619 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.848507 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 619 election: Election decided. Result: candidate lost.
I1014 19:27:30.849047 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (election complete)
I1014 19:27:30.849107 30167 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Leader election lost for term 619. Reason: could not achieve majority
I1014 19:27:30.850028 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:30.850080 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:30.850121 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 618 FOLLOWER]: Advancing to term 619
I1014 19:27:30.851936 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 619 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.852376 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 619 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:30.852531 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 619 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.853096 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 619 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:30.853271 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 619: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:30.853662 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 619 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 619: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:30.853919 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 619 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.854090 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.854997 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 619 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 619 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:30.855065 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 619 election: Election decided. Result: candidate lost.
I1014 19:27:30.855334 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (election complete)
I1014 19:27:30.855383 30382 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 619 FOLLOWER]: Leader election lost for term 619. Reason: could not achieve majority
I1014 19:27:30.858198 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 619 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 618. Current term is 619. Ops: []
I1014 19:27:30.858604 30558 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 618.476, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I1014 19:27:30.858840 30558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Stepping down as leader of term 618
I1014 19:27:30.858880 30558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:30.859067 30558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 618.476, Last appended by leader: 476, Current term: 618, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.859297 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:30.859340 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:30.859431 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 618 FOLLOWER]: Advancing to term 619
I1014 19:27:30.859566 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 618. Current term is 619. Ops: []
I1014 19:27:30.957701 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:30.957815 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:30.957893 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 619 FOLLOWER]: Advancing to term 620
I1014 19:27:30.959899 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.960446 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 620 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.960639 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 620 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.961052 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 620 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:30.961171 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 619 FOLLOWER]: Advancing to term 620
I1014 19:27:30.961236 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (vote granted)
I1014 19:27:30.961172 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 620 candidate_status { last_received { term: 618 index: 476 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:30.961311 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 619 FOLLOWER]: Advancing to term 620
I1014 19:27:30.961367 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:30.962632 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 620 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 620.
I1014 19:27:30.962662 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 620 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 620.
I1014 19:27:30.962901 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 620 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:30.962961 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 620 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:30.963018 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 620 election: Election decided. Result: candidate won.
I1014 19:27:30.963238 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.157s (election complete)
I1014 19:27:30.963304 30167 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 FOLLOWER]: Leader election won for term 620
I1014 19:27:30.963420 30167 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:30.963632 30167 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 618.476, Last appended by leader: 476, Current term: 620, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:30.964347 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.964540 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.965000 30880 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:30.966343 30880 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:30.967267 30880 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:30.967900 30880 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:30.969741 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 620 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 618 index: 476. Preceding OpId from leader: term: 620 index: 478. (index mismatch)
I1014 19:27:30.969765 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 620 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 618 index: 476. Preceding OpId from leader: term: 620 index: 478. (index mismatch)
I1014 19:27:30.970211 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I1014 19:27:30.970571 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I1014 19:27:30.971644 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.971837 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.972093 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.972291 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.973793 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.973969 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:30.974539 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.974598 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.974800 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.974800 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:30.975039 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 620 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:30.975160 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.074124 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 620 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:31.074200 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:31.074254 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 620 FOLLOWER]: Advancing to term 621
I1014 19:27:31.076007 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 620 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:31.076076 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:31.076123 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 620 FOLLOWER]: Advancing to term 621
I1014 19:27:31.076313 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.076854 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 621 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.076995 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 621 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.077762 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 621 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.077949 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 621 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.078135 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 621 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.078428 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 621 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 621 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.078619 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 621 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.078783 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 621 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.079161 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 620. Current term is 621. Ops: []
I1014 19:27:31.079164 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 621 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.079330 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 621 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 621: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:31.079661 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 621 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.079783 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 621 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 621 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 621: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:31.079850 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 621 election: Election decided. Result: candidate lost.
I1014 19:27:31.079818 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 621: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.079823 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 621 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.079972 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 621 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.080216 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 621 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 621: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.080263 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:31.080315 30382 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Leader election lost for term 621. Reason: could not achieve majority
I1014 19:27:31.080173 30882 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 620.478, Next index: 479, Last known committed idx: 478, Time since last communication: 0.000s
I1014 19:27:31.080371 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 621 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 621 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.080428 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 621 election: Election decided. Result: candidate lost.
I1014 19:27:31.080489 30167 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Stepping down as leader of term 620
I1014 19:27:31.080545 30167 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:31.080724 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:31.080785 30558 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 621 FOLLOWER]: Leader election lost for term 621. Reason: could not achieve majority
I1014 19:27:31.080744 30167 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 478, Committed index: 478, Last appended: 620.478, Last appended by leader: 478, Current term: 620, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.080938 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:31.081004 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:31.081140 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 620 FOLLOWER]: Advancing to term 621
I1014 19:27:31.181051 30382 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:31.181116 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.183s (starting election)
I1014 19:27:31.181164 30382 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 621 FOLLOWER]: Advancing to term 622
I1014 19:27:31.182538 30382 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.182894 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 622 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.183032 30382 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 622 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.183442 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 622 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.183593 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 621 FOLLOWER]: Advancing to term 622
I1014 19:27:31.183538 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 622 candidate_status { last_received { term: 620 index: 478 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.183671 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (vote granted)
I1014 19:27:31.183693 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 621 FOLLOWER]: Advancing to term 622
I1014 19:27:31.183778 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.158s (vote granted)
I1014 19:27:31.185065 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 622 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 622.
I1014 19:27:31.185247 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 622 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 622.
I1014 19:27:31.185340 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 622 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.185392 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 622 election: Election decided. Result: candidate won.
I1014 19:27:31.185519 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 622 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.185596 30382 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:31.185648 30382 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 FOLLOWER]: Leader election won for term 622
I1014 19:27:31.185926 30382 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:31.186090 30382 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 478, Committed index: 478, Last appended: 620.478, Last appended by leader: 478, Current term: 622, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.187309 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.187530 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.187880 30934 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:31.189218 30934 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:31.190129 30934 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:31.190779 30934 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:31.192520 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 622 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 620 index: 478. Preceding OpId from leader: term: 622 index: 480. (index mismatch)
I1014 19:27:31.192598 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 622 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 620 index: 478. Preceding OpId from leader: term: 622 index: 480. (index mismatch)
I1014 19:27:31.192924 30933 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 479, Last known committed idx: 478, Time since last communication: 0.000s
I1014 19:27:31.193037 30382 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 479, Last known committed idx: 478, Time since last communication: 0.000s
I1014 19:27:31.194320 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.194537 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.194756 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.194993 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.196655 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.196838 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.196817 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.196990 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.197170 30382 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.197326 30382 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.197062 30935 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 622 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.197593 30935 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.296922 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 622 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:31.297034 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:31.297123 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 622 FOLLOWER]: Advancing to term 623
I1014 19:27:31.297143 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 622 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:31.297210 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (starting election)
I1014 19:27:31.297256 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 622 FOLLOWER]: Advancing to term 623
I1014 19:27:31.299607 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.300163 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 623 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.300290 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.300688 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 623 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.300729 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 623 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.300834 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 623 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.300845 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 623 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.301031 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 623 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.301175 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 623 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.301342 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 623: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:31.301419 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 623 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.301468 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 623 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.301578 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 623 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.301746 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 623: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:31.301750 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 623 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 623 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.301889 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 623 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 623: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:31.301903 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 623 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 623 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.301946 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 623 election: Election decided. Result: candidate lost.
I1014 19:27:31.302328 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:31.302335 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 623 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 623: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:31.302407 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 623 election: Election decided. Result: candidate lost.
I1014 19:27:31.302522 30167 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Leader election lost for term 623. Reason: could not achieve majority
I1014 19:27:31.303489 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:31.303539 30558 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Leader election lost for term 623. Reason: could not achieve majority
I1014 19:27:31.307600 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 622. Current term is 623. Ops: []
I1014 19:27:31.308202 30936 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 622.480, Next index: 481, Last known committed idx: 480, Time since last communication: 0.000s
I1014 19:27:31.308465 30933 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Stepping down as leader of term 622
I1014 19:27:31.308547 30933 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:31.308729 30933 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 480, Committed index: 480, Last appended: 622.480, Last appended by leader: 480, Current term: 622, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.308903 30933 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:31.308933 30933 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:31.309036 30933 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 622 FOLLOWER]: Advancing to term 623
I1014 19:27:31.402761 30167 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:31.402899 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.215s (starting election)
I1014 19:27:31.403029 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 623 FOLLOWER]: Advancing to term 624
I1014 19:27:31.404546 30167 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.404973 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 624 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.405215 30167 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 624 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.405586 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 624 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.405705 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 623 FOLLOWER]: Advancing to term 624
I1014 19:27:31.405706 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 624 candidate_status { last_received { term: 622 index: 480 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.405772 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (vote granted)
I1014 19:27:31.405827 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 623 FOLLOWER]: Advancing to term 624
I1014 19:27:31.405884 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.167s (vote granted)
I1014 19:27:31.407161 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 624 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 624.
I1014 19:27:31.407238 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 624.
I1014 19:27:31.407438 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 624 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.407559 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 624 election: Election decided. Result: candidate won.
I1014 19:27:31.407683 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 624 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.407809 30167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.157s (election complete)
I1014 19:27:31.407904 30167 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 FOLLOWER]: Leader election won for term 624
I1014 19:27:31.408040 30167 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:31.408246 30167 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 480, Committed index: 480, Last appended: 622.480, Last appended by leader: 480, Current term: 624, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.409107 30882 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.409345 30882 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.409552 30880 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:31.410863 30880 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:31.411774 30880 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:31.412456 30880 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:31.414001 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 624 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 622 index: 480. Preceding OpId from leader: term: 624 index: 482. (index mismatch)
I1014 19:27:31.414088 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 622 index: 480. Preceding OpId from leader: term: 624 index: 482. (index mismatch)
I1014 19:27:31.414507 30882 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 481, Last known committed idx: 480, Time since last communication: 0.000s
I1014 19:27:31.414680 30167 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 481, Last known committed idx: 480, Time since last communication: 0.000s
I1014 19:27:31.416012 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.416143 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.416414 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.416646 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.417143 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.417335 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.418085 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.418224 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.418227 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.418435 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.419669 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 624 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.419900 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.693912 30933 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 624 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:31.693985 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:31.694032 30933 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 624 FOLLOWER]: Advancing to term 625
I1014 19:27:31.696025 30933 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.696485 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 625 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.696612 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 625 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.697161 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 625 candidate_status { last_received { term: 624 index: 482 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.697156 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 625 candidate_status { last_received { term: 624 index: 482 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.697342 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 625 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.697356 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 625 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.697749 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 625 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 625 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.697831 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 625 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 625 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.697901 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 625 election: Election decided. Result: candidate lost.
I1014 19:27:31.698189 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:31.698240 30933 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Leader election lost for term 625. Reason: could not achieve majority
I1014 19:27:31.708726 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 624. Current term is 625. Ops: []
I1014 19:27:31.709266 30167 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 624.482, Next index: 483, Last known committed idx: 482, Time since last communication: 0.000s
I1014 19:27:31.709496 30167 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 LEADER]: Stepping down as leader of term 624
I1014 19:27:31.709570 30167 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:31.709753 30167 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 482, Committed index: 482, Last appended: 624.482, Last appended by leader: 482, Current term: 624, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.709939 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:31.709995 30167 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:31.710145 30167 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 624 FOLLOWER]: Advancing to term 625
I1014 19:27:31.719229 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:31.719297 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.133s (starting election)
I1014 19:27:31.719341 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 624 FOLLOWER]: Advancing to term 625
I1014 19:27:31.721314 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.721782 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 625 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.721927 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 625 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.722393 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 625 candidate_status { last_received { term: 624 index: 482 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.722496 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 625 candidate_status { last_received { term: 624 index: 482 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.722579 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (vote granted)
I1014 19:27:31.722744 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 625: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.723120 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 625 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 625: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.724115 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 625.
I1014 19:27:31.724366 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 625 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.724416 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 625 election: Election decided. Result: candidate won.
I1014 19:27:31.724609 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:31.724669 30558 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 FOLLOWER]: Leader election won for term 625
I1014 19:27:31.724948 30558 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:31.725148 30558 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 482, Committed index: 482, Last appended: 624.482, Last appended by leader: 482, Current term: 625, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.726373 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.726579 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.727887 31120 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:31.729627 31120 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:31.730867 31120 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:31.731676 31120 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:31.733633 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 624 index: 482. Preceding OpId from leader: term: 625 index: 484. (index mismatch)
I1014 19:27:31.733777 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 624 index: 482. Preceding OpId from leader: term: 625 index: 484. (index mismatch)
I1014 19:27:31.734300 30558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 483, Last known committed idx: 482, Time since last communication: 0.000s
I1014 19:27:31.734691 31119 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 483, Last known committed idx: 482, Time since last communication: 0.000s
I1014 19:27:31.736582 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.736837 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.737651 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.738054 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.738142 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.738358 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.739177 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.739426 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.739902 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.740077 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.741791 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 625 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.741966 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.839066 30933 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:31.839148 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:31.839265 30933 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 625 FOLLOWER]: Advancing to term 626
I1014 19:27:31.841157 30933 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.841634 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 626 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:31.841812 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 626 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.842200 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 626 candidate_status { last_received { term: 625 index: 484 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:31.842389 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 626 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.842370 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 626 candidate_status { last_received { term: 625 index: 484 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.842592 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 626 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.842854 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 626 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 626 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.842934 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 626 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 626 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:31.843014 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 626 election: Election decided. Result: candidate lost.
I1014 19:27:31.843240 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:31.843315 30933 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Leader election lost for term 626. Reason: could not achieve majority
I1014 19:27:31.859649 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 625. Current term is 626. Ops: []
I1014 19:27:31.860319 30558 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 625.484, Next index: 485, Last known committed idx: 484, Time since last communication: 0.000s
I1014 19:27:31.860594 30558 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 LEADER]: Stepping down as leader of term 625
I1014 19:27:31.860638 30558 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:31.860819 30558 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 484, Committed index: 484, Last appended: 625.484, Last appended by leader: 484, Current term: 625, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.860999 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:31.861039 30558 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:31.861166 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 625 FOLLOWER]: Advancing to term 626
I1014 19:27:31.941293 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:31.941393 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:31.941494 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 625 FOLLOWER]: Advancing to term 626
I1014 19:27:31.943327 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.943753 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 626 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:31.943933 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 626 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.944416 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 626 candidate_status { last_received { term: 625 index: 484 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:31.944515 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 626 candidate_status { last_received { term: 625 index: 484 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:31.944612 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 626: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.944649 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.101s (vote granted)
I1014 19:27:31.944967 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 626 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 626: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:31.946640 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 626 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 626.
I1014 19:27:31.946954 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 626 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:31.947012 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 626 election: Election decided. Result: candidate won.
I1014 19:27:31.947257 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:31.947343 30735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 FOLLOWER]: Leader election won for term 626
I1014 19:27:31.947508 30735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:31.947691 30735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 484, Committed index: 484, Last appended: 625.484, Last appended by leader: 484, Current term: 626, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:31.948611 30167 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.948803 30167 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.949265 31182 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:31.950743 31182 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:31.951932 31182 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:31.952903 31182 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:31.954948 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 626 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 625 index: 484. Preceding OpId from leader: term: 626 index: 486. (index mismatch)
I1014 19:27:31.954947 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 625 index: 484. Preceding OpId from leader: term: 626 index: 486. (index mismatch)
I1014 19:27:31.955458 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 485, Last known committed idx: 484, Time since last communication: 0.000s
I1014 19:27:31.955688 30167 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 485, Last known committed idx: 484, Time since last communication: 0.000s
I1014 19:27:31.957381 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.957612 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.957736 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.957950 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.959072 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.959267 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.959367 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.959573 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.959772 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.959712 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 626 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:31.959970 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:31.960002 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:31.959985 25847 mvcc.cc:203] Tried to move safe_time back from 6306080571202727936 to 6306080571179892736. Current Snapshot: MvccSnapshot[committed={T|T < 6306080571202727936 or (T in {6306080571202727936})}]
I1014 19:27:32.143103 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 626 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:32.143179 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:32.143230 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 626 FOLLOWER]: Advancing to term 627
I1014 19:27:32.145181 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.145556 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 627 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.145675 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 627 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.146126 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 627 candidate_status { last_received { term: 626 index: 486 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.146173 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 627 candidate_status { last_received { term: 626 index: 486 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.146320 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 627 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.146323 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 627 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.146622 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 627 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 627 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.146757 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 627 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 627 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.146809 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 627 election: Election decided. Result: candidate lost.
I1014 19:27:32.146992 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:27:32.147048 30558 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Leader election lost for term 627. Reason: could not achieve majority
I1014 19:27:32.153036 30933 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:32.153112 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (starting election)
I1014 19:27:32.153182 30933 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 626 FOLLOWER]: Advancing to term 627
I1014 19:27:32.154583 30933 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.154983 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 627 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.155190 30933 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 627 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.155339 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 626. Current term is 627. Ops: []
I1014 19:27:32.155843 31183 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 626.486, Next index: 487, Last known committed idx: 486, Time since last communication: 0.000s
I1014 19:27:32.156224 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 627 candidate_status { last_received { term: 626 index: 486 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.156353 31183 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 LEADER]: Stepping down as leader of term 626
I1014 19:27:32.156397 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 627: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:32.156420 31183 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:32.156783 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 627 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 627: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:32.156675 31183 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 486, Committed index: 486, Last appended: 626.486, Last appended by leader: 486, Current term: 626, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.157016 31183 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:32.157111 31183 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:32.157268 31183 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 626 FOLLOWER]: Advancing to term 627
I1014 19:27:32.157565 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 627 candidate_status { last_received { term: 626 index: 486 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.158947 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.104s (vote granted)
I1014 19:27:32.160564 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 627 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 627.
I1014 19:27:32.161015 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 627 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.161123 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 627 election: Election decided. Result: candidate won.
I1014 19:27:32.161345 30933 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:32.161438 30933 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 FOLLOWER]: Leader election won for term 627
I1014 19:27:32.161733 30933 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:32.161932 30933 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 486, Committed index: 486, Last appended: 626.486, Last appended by leader: 486, Current term: 627, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.162817 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.163084 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.163620 31284 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:32.164974 31284 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:32.165926 31284 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:32.166785 31284 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:32.168457 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 626 index: 486. Preceding OpId from leader: term: 627 index: 488. (index mismatch)
I1014 19:27:32.168494 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 627 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 626 index: 486. Preceding OpId from leader: term: 627 index: 488. (index mismatch)
I1014 19:27:32.168941 30933 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 487, Last known committed idx: 486, Time since last communication: 0.000s
I1014 19:27:32.169121 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 487, Last known committed idx: 486, Time since last communication: 0.000s
I1014 19:27:32.170980 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.171169 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.171492 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.171697 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.173099 30933 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.173331 30933 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.173327 30558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.173539 30558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.174422 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.174690 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.176960 31285 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 627 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.177443 31285 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.274763 31183 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 627 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:32.274853 31183 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:32.274924 31183 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 627 FOLLOWER]: Advancing to term 628
I1014 19:27:32.276351 31183 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.276787 31183 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 628 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.276983 31183 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 628 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.277338 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 628 candidate_status { last_received { term: 627 index: 488 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.277530 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 628 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.277467 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 628 candidate_status { last_received { term: 627 index: 488 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.277618 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 628 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.277837 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 628 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 628 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.277990 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 628 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 628 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.278053 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 628 election: Election decided. Result: candidate lost.
I1014 19:27:32.278228 31183 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:32.278293 31183 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Leader election lost for term 628. Reason: could not achieve majority
I1014 19:27:32.282380 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 627. Current term is 628. Ops: []
I1014 19:27:32.282897 31285 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 627.488, Next index: 489, Last known committed idx: 488, Time since last communication: 0.000s
I1014 19:27:32.283120 31283 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 LEADER]: Stepping down as leader of term 627
I1014 19:27:32.283185 31283 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:32.283411 31283 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 488, Committed index: 488, Last appended: 627.488, Last appended by leader: 488, Current term: 627, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.283643 31283 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:32.283730 31283 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:32.283885 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 627 FOLLOWER]: Advancing to term 628
I1014 19:27:32.354439 30558 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:32.354545 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:32.354604 30558 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 627 FOLLOWER]: Advancing to term 628
I1014 19:27:32.356076 30558 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.356456 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 628 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.356611 30558 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 628 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.357117 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 628 candidate_status { last_received { term: 627 index: 488 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.357125 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 628 candidate_status { last_received { term: 627 index: 488 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.357270 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 628: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:32.357312 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:32.357564 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 628 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 628: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:32.358680 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 628 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 628.
I1014 19:27:32.358928 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 628 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.358968 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 628 election: Election decided. Result: candidate won.
I1014 19:27:32.359119 30558 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:32.359184 30558 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 FOLLOWER]: Leader election won for term 628
I1014 19:27:32.359266 30558 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:32.359438 30558 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 488, Committed index: 488, Last appended: 627.488, Last appended by leader: 488, Current term: 628, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.360245 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.360460 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.360906 31342 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:32.362264 31342 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:32.363216 31342 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:32.363837 31342 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:32.365713 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 627 index: 488. Preceding OpId from leader: term: 628 index: 490. (index mismatch)
I1014 19:27:32.365772 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 628 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 627 index: 488. Preceding OpId from leader: term: 628 index: 490. (index mismatch)
I1014 19:27:32.366338 30558 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 489, Last known committed idx: 488, Time since last communication: 0.000s
I1014 19:27:32.366551 31119 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 489, Last known committed idx: 488, Time since last communication: 0.000s
I1014 19:27:32.368263 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.368527 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.368471 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.368695 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.369418 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.369683 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.370213 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.370178 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.370391 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.370458 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.370606 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 628 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.370745 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.470408 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 628 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:32.470515 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:32.470578 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 628 FOLLOWER]: Advancing to term 629
I1014 19:27:32.472643 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.473089 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 629 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.473280 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 629 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.473709 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 629 candidate_status { last_received { term: 628 index: 490 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.473795 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 629 candidate_status { last_received { term: 628 index: 490 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.473915 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 629 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.473975 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 629 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.474282 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 629 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 629 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.474448 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 629 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 629 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.474534 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 629 election: Election decided. Result: candidate lost.
I1014 19:27:32.474757 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:32.474819 31283 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Leader election lost for term 629. Reason: could not achieve majority
I1014 19:27:32.484397 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 628. Current term is 629. Ops: []
I1014 19:27:32.484987 31343 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 628.490, Next index: 491, Last known committed idx: 490, Time since last communication: 0.000s
I1014 19:27:32.485357 31119 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 LEADER]: Stepping down as leader of term 628
I1014 19:27:32.485406 31119 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:32.485688 31119 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 490, Committed index: 490, Last appended: 628.490, Last appended by leader: 490, Current term: 628, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.485919 31119 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:32.485958 31119 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:32.486109 31119 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 628 FOLLOWER]: Advancing to term 629
I1014 19:27:32.560904 31183 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:32.560995 31183 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:32.561069 31183 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 628 FOLLOWER]: Advancing to term 629
I1014 19:27:32.562531 31183 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.562952 31183 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 629 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.563151 31183 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 629 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.563652 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 629 candidate_status { last_received { term: 628 index: 490 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.563691 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 629 candidate_status { last_received { term: 628 index: 490 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.563813 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:32.563854 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 629: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:32.564184 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 629 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 629: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:32.565280 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 629.
I1014 19:27:32.565629 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 629 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.565728 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 629 election: Election decided. Result: candidate won.
I1014 19:27:32.565984 31183 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:32.566061 31183 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 FOLLOWER]: Leader election won for term 629
I1014 19:27:32.566160 31183 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:32.566395 31183 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 490, Committed index: 490, Last appended: 628.490, Last appended by leader: 490, Current term: 629, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.567271 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.567440 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.567864 31390 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:32.569250 31390 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:32.570230 31390 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:32.570909 31390 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:32.572589 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 628 index: 490. Preceding OpId from leader: term: 629 index: 492. (index mismatch)
I1014 19:27:32.572636 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 628 index: 490. Preceding OpId from leader: term: 629 index: 492. (index mismatch)
I1014 19:27:32.573117 31183 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 491, Last known committed idx: 490, Time since last communication: 0.000s
I1014 19:27:32.573315 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 491, Last known committed idx: 490, Time since last communication: 0.000s
I1014 19:27:32.574820 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.575114 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.575399 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.575670 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.576447 31391 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.576818 31391 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.576860 31392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.577132 31392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.577124 25848 mvcc.cc:203] Tried to move safe_time back from 6306080573733363712 to 6306080573714251776. Current Snapshot: MvccSnapshot[committed={T|T < 6306080573733363712 or (T in {6306080573733363712})}]
I1014 19:27:32.577035 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.577265 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.578150 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 629 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.578652 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.642208 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:32.843340 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:32.843457 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:32.843531 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 629 FOLLOWER]: Advancing to term 630
I1014 19:27:32.845710 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 630 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.846200 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 630 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.846423 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 630 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.846832 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 630 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.847056 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 630 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.847458 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 630 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 630 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.847493 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 630 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.847694 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 630 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.848315 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 630 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 630 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.848392 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 630 election: Election decided. Result: candidate lost.
I1014 19:27:32.858311 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:32.858394 31283 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 630 FOLLOWER]: Leader election lost for term 630. Reason: could not achieve majority
I1014 19:27:32.859277 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 630 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 629. Current term is 630. Ops: []
I1014 19:27:32.859844 31183 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 629.492, Next index: 493, Last known committed idx: 492, Time since last communication: 0.000s
I1014 19:27:32.860141 31183 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 LEADER]: Stepping down as leader of term 629
I1014 19:27:32.860199 31183 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:32.860416 31183 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 492, Committed index: 492, Last appended: 629.492, Last appended by leader: 492, Current term: 629, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.860641 31183 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:32.860693 31183 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:32.860837 31183 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 629 FOLLOWER]: Advancing to term 630
I1014 19:27:32.940796 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 630 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:32.940886 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:32.940963 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 630 FOLLOWER]: Advancing to term 631
I1014 19:27:32.942430 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.942878 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 631 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.943049 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 631 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:32.943433 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 631 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.943610 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 630 FOLLOWER]: Advancing to term 631
I1014 19:27:32.943557 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 631 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:32.943686 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.176s (vote granted)
I1014 19:27:32.943719 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 631 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.944017 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 631 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 631 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.945658 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 631.
I1014 19:27:32.946040 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 631 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.946120 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 631 election: Election decided. Result: candidate won.
I1014 19:27:32.946339 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.135s (election complete)
I1014 19:27:32.946452 30735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 FOLLOWER]: Leader election won for term 631
I1014 19:27:32.946588 30735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:32.946787 30735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 492, Committed index: 492, Last appended: 629.492, Last appended by leader: 492, Current term: 631, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.947682 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 631 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.947899 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.948107 31390 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:32.949590 31390 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:32.950631 31390 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:32.951300 31390 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:32.953604 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 629 FOLLOWER]: Advancing to term 631
I1014 19:27:32.953806 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 629 index: 492. Preceding OpId from leader: term: 631 index: 494. (index mismatch)
I1014 19:27:32.954356 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 493, Last known committed idx: 492, Time since last communication: 0.000s
I1014 19:27:32.955370 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 631 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 629 index: 492. Preceding OpId from leader: term: 631 index: 494. (index mismatch)
I1014 19:27:32.955844 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 493, Last known committed idx: 492, Time since last communication: 0.000s
I1014 19:27:32.956406 31119 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 631 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:32.956487 31119 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:32.956538 31119 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 631 FOLLOWER]: Advancing to term 632
I1014 19:27:32.956982 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 631 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.957165 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.957733 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 631 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.957949 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:32.958359 31119 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 632 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.958791 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 632 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 631. Current term is 632. Ops: []
I1014 19:27:32.958925 31119 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 632 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:32.959112 31119 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 632 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:32.959578 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 632 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:32.959616 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 632 candidate_status { last_received { term: 629 index: 492 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:32.959619 30735 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 629.492, Next index: 493, Last known committed idx: 492, Time since last communication: 0.000s
I1014 19:27:32.959985 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 632 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.959926 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 631 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:32.960091 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:32.960091 31392 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 LEADER]: Stepping down as leader of term 631
I1014 19:27:32.960180 31392 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:32.960209 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 632 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:32.960378 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 632 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 632 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:32.960361 31392 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 494, Committed index: 494, Last appended: 631.494, Last appended by leader: 494, Current term: 631, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:32.960577 31392 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:32.960630 31392 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:32.960731 31392 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 631 FOLLOWER]: Advancing to term 632
I1014 19:27:32.967650 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 632 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 632 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:32.967725 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 632 election: Election decided. Result: candidate lost.
I1014 19:27:32.967952 31119 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:32.968020 31119 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 632 FOLLOWER]: Leader election lost for term 632. Reason: could not achieve majority
I1014 19:27:33.053467 31392 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 632 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:33.053556 31392 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:33.053625 31392 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 632 FOLLOWER]: Advancing to term 633
I1014 19:27:33.055057 31392 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.055421 31392 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 633 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.055621 31392 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 633 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.056071 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 633 candidate_status { last_received { term: 631 index: 494 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.056231 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 633 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.056239 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 633 candidate_status { last_received { term: 631 index: 494 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.056363 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 632 FOLLOWER]: Advancing to term 633
I1014 19:27:33.056433 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:33.056584 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 633 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 633 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.057862 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 633.
I1014 19:27:33.058153 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 633 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.058224 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 633 election: Election decided. Result: candidate won.
I1014 19:27:33.058517 31392 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (election complete)
I1014 19:27:33.058593 31392 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 FOLLOWER]: Leader election won for term 633
I1014 19:27:33.058764 31392 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:33.059027 31392 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 494, Committed index: 494, Last appended: 631.494, Last appended by leader: 494, Current term: 633, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.059514 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:33.059574 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:33.059612 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 631 FOLLOWER]: Advancing to term 632
I1014 19:27:33.060096 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.060472 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.060672 31390 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:33.061113 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 632 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.061431 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 632 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.061569 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 632 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.062041 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 632 candidate_status { last_received { term: 631 index: 494 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.062166 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 632. Current term is 633.
W1014 19:27:33.062481 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 632 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 632. Current term is 633.
I1014 19:27:33.062510 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 632 election: Cancelling election due to peer responding with higher term
I1014 19:27:33.062680 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:33.062732 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 632 FOLLOWER]: Advancing to term 633
I1014 19:27:33.062908 31390 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:33.062985 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 632 candidate_status { last_received { term: 631 index: 494 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.063238 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 632 because replica is either leader or believes a valid leader to be alive.
W1014 19:27:33.063575 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 632 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 632 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.064025 31283 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 633 FOLLOWER]: Leader election lost for term 632. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 632. Current term is 633.
I1014 19:27:33.064208 31390 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:33.064937 31390 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:33.066619 28826 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 633 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 631 index: 494. Preceding OpId from leader: term: 633 index: 496. (index mismatch)
I1014 19:27:33.066629 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 629 index: 492. Preceding OpId from leader: term: 633 index: 496. (index mismatch)
I1014 19:27:33.067045 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 495, Last known committed idx: 492, Time since last communication: 0.000s
I1014 19:27:33.067286 31392 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 495, Last known committed idx: 494, Time since last communication: 0.000s
I1014 19:27:33.069090 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.069406 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.070087 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.070247 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.070582 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.070785 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.070698 31119 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.070960 31119 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.071045 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.071202 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.071719 31392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.071934 31392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.072305 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 633 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.072523 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.170024 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 633 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:33.170107 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:33.170157 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 633 FOLLOWER]: Advancing to term 634
I1014 19:27:33.172199 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.172688 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 634 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.172832 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 634 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.173300 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:33.173375 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (starting election)
I1014 19:27:33.173452 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 633 FOLLOWER]: Advancing to term 634
I1014 19:27:33.173396 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 634 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.173410 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 634 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.173629 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 634 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.174103 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 634 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 634 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.175050 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.175308 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 634: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.175444 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 634 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.175582 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 634 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.175770 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 634 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 634: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.175814 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 634 election: Election decided. Result: candidate lost.
I1014 19:27:33.175956 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:33.176004 31283 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Leader election lost for term 634. Reason: could not achieve majority
I1014 19:27:33.176048 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 634 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.176203 28826 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 634: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:33.176651 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 634 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 634: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:33.176702 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 634 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.176841 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 634 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.177119 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 634 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 634 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.177186 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 634 election: Election decided. Result: candidate lost.
I1014 19:27:33.177407 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:33.177485 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Leader election lost for term 634. Reason: could not achieve majority
I1014 19:27:33.180172 28826 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 633. Current term is 634. Ops: []
I1014 19:27:33.180691 31392 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 633.496, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I1014 19:27:33.180912 30735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Stepping down as leader of term 633
I1014 19:27:33.180974 30735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:33.181200 30735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 633.496, Last appended by leader: 496, Current term: 633, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.181416 30735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:33.181499 30735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:33.181689 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 633 FOLLOWER]: Advancing to term 634
I1014 19:27:33.294268 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 634 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:33.294368 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.101s (starting election)
I1014 19:27:33.294443 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 634 FOLLOWER]: Advancing to term 635
I1014 19:27:33.296480 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:33.296561 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:33.296618 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 634 FOLLOWER]: Advancing to term 635
I1014 19:27:33.296656 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.297112 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 635 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.297257 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 635 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.297791 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 635 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.297847 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 635 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.297920 28826 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 634 FOLLOWER]: Advancing to term 635
I1014 19:27:33.297973 28826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (vote granted)
I1014 19:27:33.298295 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.298564 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 635: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.298725 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 635 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.298859 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 635 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.299023 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 635 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 635: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.299497 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 635 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.299693 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 635 candidate_status { last_received { term: 633 index: 496 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.299844 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 635: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:33.299836 28826 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 635.
I1014 19:27:33.300127 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 635 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 635: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:33.300189 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 635 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.300246 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 635 election: Election decided. Result: candidate won.
I1014 19:27:33.300546 28825 raft_consensus.cc:2230] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 635 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:33.300601 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:33.300674 30735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 FOLLOWER]: Leader election won for term 635
I1014 19:27:33.300814 30735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:33.301000 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 635 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Service unavailable: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 635 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:33.301057 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 635 election: Election decided. Result: candidate lost.
I1014 19:27:33.301218 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.196s (election complete)
I1014 19:27:33.301270 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Leader election lost for term 635. Reason: could not achieve majority
I1014 19:27:33.301414 30735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 633.496, Last appended by leader: 496, Current term: 635, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.302367 31392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.302600 31392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.302816 31390 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:33.304721 31390 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:33.306080 31390 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:33.306969 31390 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:33.308840 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 633 index: 496. Preceding OpId from leader: term: 635 index: 498. (index mismatch)
I1014 19:27:33.308914 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 633 index: 496. Preceding OpId from leader: term: 635 index: 498. (index mismatch)
I1014 19:27:33.309360 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I1014 19:27:33.309648 31392 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I1014 19:27:33.311152 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.311344 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.311631 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.311818 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.312304 31392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.312305 31183 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.312603 31392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.312608 31183 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.313205 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.313370 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.314095 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 635 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.314273 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.413625 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:33.413700 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:33.413755 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 635 FOLLOWER]: Advancing to term 636
I1014 19:27:33.415837 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.416266 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 636 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.416411 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 636 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.416925 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 636 candidate_status { last_received { term: 635 index: 498 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.417143 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 636 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.417481 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 636 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 636 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.417824 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 636 candidate_status { last_received { term: 635 index: 498 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.417995 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 636 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.418285 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 636 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 636 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.418344 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 636 election: Election decided. Result: candidate lost.
I1014 19:27:33.418701 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:33.418756 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election lost for term 636. Reason: could not achieve majority
I1014 19:27:33.422016 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 635. Current term is 636. Ops: []
I1014 19:27:33.422581 31583 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 635.498, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I1014 19:27:33.422788 30735 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 LEADER]: Stepping down as leader of term 635
I1014 19:27:33.422842 30735 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:33.423012 30735 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 635.498, Last appended by leader: 498, Current term: 635, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.423211 30735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:33.423262 30735 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:33.423398 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 635 FOLLOWER]: Advancing to term 636
I1014 19:27:33.491902 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:33.491972 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:33.492014 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 635 FOLLOWER]: Advancing to term 636
I1014 19:27:33.494107 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.494596 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 636 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.494745 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 636 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.495185 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 636 candidate_status { last_received { term: 635 index: 498 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.495276 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 636 candidate_status { last_received { term: 635 index: 498 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.495347 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (vote granted)
I1014 19:27:33.495465 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 636: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.495868 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 636 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 636: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.497318 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 636 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 636.
I1014 19:27:33.497655 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 636 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.497711 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 636 election: Election decided. Result: candidate won.
I1014 19:27:33.497890 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (election complete)
I1014 19:27:33.497947 31283 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 FOLLOWER]: Leader election won for term 636
I1014 19:27:33.498221 31283 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:33.498391 31283 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 635.498, Last appended by leader: 498, Current term: 636, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.499649 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.499845 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.500398 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:33.502174 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:33.503417 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:33.504261 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:33.506345 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 635 index: 498. Preceding OpId from leader: term: 636 index: 500. (index mismatch)
I1014 19:27:33.506485 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 636 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 635 index: 498. Preceding OpId from leader: term: 636 index: 500. (index mismatch)
I1014 19:27:33.506969 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I1014 19:27:33.507143 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I1014 19:27:33.508684 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.508709 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.508870 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.508886 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.510901 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.511081 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.512039 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.512029 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.512217 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.512218 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.512465 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 636 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.512631 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.611863 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 636 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:33.611943 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:33.611996 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 636 FOLLOWER]: Advancing to term 637
I1014 19:27:33.614150 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 637 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.614679 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 637 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.614830 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 637 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.615885 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 637 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.616042 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.615962 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 637 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.616138 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.616367 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 637 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.616724 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 637 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.616787 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 637 election: Election decided. Result: candidate lost.
I1014 19:27:33.616966 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:33.617022 30735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 637 FOLLOWER]: Leader election lost for term 637. Reason: could not achieve majority
I1014 19:27:33.623653 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 637 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 636. Current term is 637. Ops: []
I1014 19:27:33.624109 31642 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 636.500, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I1014 19:27:33.624334 31644 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 LEADER]: Stepping down as leader of term 636
I1014 19:27:33.624384 31644 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:33.624588 31644 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 636.500, Last appended by leader: 500, Current term: 636, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.624783 31644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:33.624814 31644 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:33.624915 31644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 636 FOLLOWER]: Advancing to term 637
I1014 19:27:33.703595 31644 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 637 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:33.703666 31644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:33.703722 31644 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 637 FOLLOWER]: Advancing to term 638
I1014 19:27:33.705050 31644 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.705389 31644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 638 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.705502 31644 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 638 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.706020 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 638 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.706022 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 638 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.706146 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 637 FOLLOWER]: Advancing to term 638
I1014 19:27:33.706176 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 638 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.706209 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.183s (vote granted)
I1014 19:27:33.706512 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 638 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 638 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.707600 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 638.
I1014 19:27:33.707878 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 638 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.707932 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 638 election: Election decided. Result: candidate won.
I1014 19:27:33.708082 31644 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:33.708130 31644 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 FOLLOWER]: Leader election won for term 638
I1014 19:27:33.708201 31644 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:33.708354 31644 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 636.500, Last appended by leader: 500, Current term: 638, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.709110 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.709277 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.709460 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:33.710750 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:33.711482 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:33.711572 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (starting election)
I1014 19:27:33.711622 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 636 FOLLOWER]: Advancing to term 637
I1014 19:27:33.711781 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:33.712582 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:33.713624 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 637 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.714099 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 637 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.714241 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 637 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.714884 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 637 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.715051 28640 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 637. Current term is 638.
I1014 19:27:33.715113 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 637 FOLLOWER]: Advancing to term 638
W1014 19:27:33.715363 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 637 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 637. Current term is 638.
I1014 19:27:33.715392 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 637 election: Cancelling election due to peer responding with higher term
I1014 19:27:33.715500 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 637 candidate_status { last_received { term: 636 index: 500 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.715654 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.715833 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 636 index: 500. Preceding OpId from leader: term: 638 index: 502. (index mismatch)
W1014 19:27:33.716141 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 637 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 637 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.716235 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I1014 19:27:33.717150 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 638 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 636 index: 500. Preceding OpId from leader: term: 638 index: 502. (index mismatch)
I1014 19:27:33.717258 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:33.717316 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 638 FOLLOWER]: Leader election lost for term 637. Reason: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 637. Current term is 638.
I1014 19:27:33.717674 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I1014 19:27:33.717885 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.718058 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.719175 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.719362 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.719485 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.719689 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.720337 31644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.720541 31644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.721233 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.721398 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.721743 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 638 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.721928 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.820585 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:33.820663 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:33.820711 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 638 FOLLOWER]: Advancing to term 639
I1014 19:27:33.822752 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.822950 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 638 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:33.823015 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:33.823071 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 638 FOLLOWER]: Advancing to term 639
I1014 19:27:33.823181 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 639 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.823318 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 639 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.824020 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 639 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.824198 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 639 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.824460 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 639 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.824558 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 639 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 639 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.824692 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 639 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.825137 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 639 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.825273 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 639 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:33.826000 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 639 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:33.826146 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 639 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.826314 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 639 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.826426 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 639: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:33.826553 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 639 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 639 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:33.826706 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 639 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 639: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:33.826761 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 639 election: Election decided. Result: candidate lost.
I1014 19:27:33.827076 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:33.827131 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 639 FOLLOWER]: Leader election lost for term 639. Reason: could not achieve majority
I1014 19:27:33.827281 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 639 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 639: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.827692 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 639 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 639 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 639: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:33.827755 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 639 election: Election decided. Result: candidate lost.
I1014 19:27:33.828032 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:33.828078 30735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Leader election lost for term 639. Reason: could not achieve majority
I1014 19:27:33.829367 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 638. Current term is 639. Ops: []
I1014 19:27:33.829843 31642 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 638.502, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I1014 19:27:33.830050 31642 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Stepping down as leader of term 638
I1014 19:27:33.830085 31642 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:33.830286 31642 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 638.502, Last appended by leader: 502, Current term: 638, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.830493 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:33.830528 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:33.830634 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 638 FOLLOWER]: Advancing to term 639
I1014 19:27:33.912478 31642 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 639 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:33.912559 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:33.912609 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 639 FOLLOWER]: Advancing to term 640
I1014 19:27:33.914572 31642 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.915069 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 640 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.915295 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 640 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.915881 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 640 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:33.916007 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 639 FOLLOWER]: Advancing to term 640
I1014 19:27:33.916064 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.172s (vote granted)
I1014 19:27:33.917922 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 640 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 640.
I1014 19:27:33.918273 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 640 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:33.918334 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 640 election: Election decided. Result: candidate won.
I1014 19:27:33.918514 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:33.918601 31642 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 FOLLOWER]: Leader election won for term 640
I1014 19:27:33.918709 31642 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:33.918792 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 640 candidate_status { last_received { term: 638 index: 502 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:33.918920 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 639 FOLLOWER]: Advancing to term 640
I1014 19:27:33.919003 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.165s (vote granted)
I1014 19:27:33.918908 31642 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 638.502, Last appended by leader: 502, Current term: 640, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:33.920060 31644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.920295 31644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.920472 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:33.920840 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 640.
I1014 19:27:33.921126 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 640 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:33.922089 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:33.923311 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:33.924083 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:33.925948 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 638 index: 502. Preceding OpId from leader: term: 640 index: 504. (index mismatch)
I1014 19:27:33.926131 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 640 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 638 index: 502. Preceding OpId from leader: term: 640 index: 504. (index mismatch)
I1014 19:27:33.926482 31644 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I1014 19:27:33.926651 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I1014 19:27:33.928726 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.928923 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.928887 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.929051 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.931100 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.931272 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.931557 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.931740 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:33.932335 31644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.932471 31644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:33.932387 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 640 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:33.932586 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.194267 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 640 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.194339 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:34.194384 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 640 FOLLOWER]: Advancing to term 641
I1014 19:27:34.196437 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.196941 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 641 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.197096 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 641 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.197705 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 641 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.197916 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 641 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.197852 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 641 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.198029 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 641 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.198328 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 641 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 641 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.198614 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 641 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 641 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.198691 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 641 election: Election decided. Result: candidate lost.
I1014 19:27:34.198890 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:34.198954 30735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Leader election lost for term 641. Reason: could not achieve majority
I1014 19:27:34.204927 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 640. Current term is 641. Ops: []
I1014 19:27:34.205627 31642 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 640.504, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I1014 19:27:34.205881 31642 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 LEADER]: Stepping down as leader of term 640
I1014 19:27:34.205921 31642 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.206101 31642 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 504, Committed index: 504, Last appended: 640.504, Last appended by leader: 504, Current term: 640, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.206286 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:34.206329 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:34.206451 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 640 FOLLOWER]: Advancing to term 641
I1014 19:27:34.287514 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.287596 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:34.287647 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 640 FOLLOWER]: Advancing to term 641
I1014 19:27:34.289466 31642 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 641 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:34.289532 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:34.289580 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 641 FOLLOWER]: Advancing to term 642
I1014 19:27:34.289758 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 641 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.290215 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 641 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.290357 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 641 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.291014 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 641 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.291088 31642 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.291280 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 641 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.291489 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 641: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:34.291556 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 642 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.291697 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 642 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.291870 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 641 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 641: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:34.292135 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 641. Current term is 642.
I1014 19:27:34.292219 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 642 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.292356 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 641 FOLLOWER]: Advancing to term 642
I1014 19:27:34.292415 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (vote granted)
I1014 19:27:34.292479 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 642 candidate_status { last_received { term: 640 index: 504 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.292583 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 641 FOLLOWER]: Advancing to term 642
I1014 19:27:34.292630 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.213s (vote granted)
I1014 19:27:34.294379 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 642 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 642.
I1014 19:27:34.294530 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 642 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 642.
I1014 19:27:34.294742 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 642 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.294795 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 642 election: Election decided. Result: candidate won.
I1014 19:27:34.295018 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:34.295078 31642 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Leader election won for term 642
I1014 19:27:34.295159 31642 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.295305 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 642 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
W1014 19:27:34.295393 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 641 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 641. Current term is 642.
I1014 19:27:34.295445 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 641 election: Cancelling election due to peer responding with higher term
I1014 19:27:34.295347 31642 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 504, Committed index: 504, Last appended: 640.504, Last appended by leader: 504, Current term: 642, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.295807 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:34.295853 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 642 FOLLOWER]: Leader election lost for term 641. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 641. Current term is 642.
I1014 19:27:34.296797 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.296980 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.297168 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:34.299116 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:34.300505 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:34.301362 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:34.303639 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 642 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 640 index: 504. Preceding OpId from leader: term: 642 index: 506. (index mismatch)
I1014 19:27:34.303642 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 642 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 640 index: 504. Preceding OpId from leader: term: 642 index: 506. (index mismatch)
I1014 19:27:34.304126 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I1014 19:27:34.304296 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 504, Time since last communication: 0.000s
I1014 19:27:34.305744 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.305985 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.306337 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.306573 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.307088 31644 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.307281 31644 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.309828 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.310060 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.310791 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.310948 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.311669 31802 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 642 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.311903 31802 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.408270 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 642 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.408350 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:34.408403 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 642 FOLLOWER]: Advancing to term 643
I1014 19:27:34.410600 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 643 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.411101 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 643 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.411259 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 643 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.411442 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 642 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.411527 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:34.411573 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 642 FOLLOWER]: Advancing to term 643
I1014 19:27:34.411972 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 643 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.412299 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 643 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.412490 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 643 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.412825 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 643 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 643 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.413635 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.413945 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 643: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:34.414124 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 643 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.414254 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 643 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.414400 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 643 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 643: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:34.414463 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 643 election: Election decided. Result: candidate lost.
I1014 19:27:34.414932 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 643 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.415071 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 643 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.415207 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 643 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.415379 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 643 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 643: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:34.415380 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 643 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 643 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.417315 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:34.417402 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 643 FOLLOWER]: Leader election lost for term 643. Reason: could not achieve majority
I1014 19:27:34.426097 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 643 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 643 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 643: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:34.426230 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 643 election: Election decided. Result: candidate lost.
I1014 19:27:34.426440 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:34.426532 30735 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Leader election lost for term 643. Reason: could not achieve majority
I1014 19:27:34.426857 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 642. Current term is 643. Ops: []
I1014 19:27:34.427363 31802 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 642.506, Next index: 507, Last known committed idx: 506, Time since last communication: 0.000s
I1014 19:27:34.427621 31283 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Stepping down as leader of term 642
I1014 19:27:34.427672 31283 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.427866 31283 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 506, Committed index: 506, Last appended: 642.506, Last appended by leader: 506, Current term: 642, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.428056 31283 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:34.428105 31283 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:34.428236 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 642 FOLLOWER]: Advancing to term 643
I1014 19:27:34.526952 31283 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 643 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:34.527021 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:34.527081 31283 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 643 FOLLOWER]: Advancing to term 644
I1014 19:27:34.528385 31283 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.528808 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 644 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.528950 31283 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 644 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.529364 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 644 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.529458 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 644 candidate_status { last_received { term: 642 index: 506 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.529530 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 643 FOLLOWER]: Advancing to term 644
I1014 19:27:34.529587 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.189s (vote granted)
I1014 19:27:34.529593 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 643 FOLLOWER]: Advancing to term 644
I1014 19:27:34.529647 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.151s (vote granted)
I1014 19:27:34.530946 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 644 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 644.
I1014 19:27:34.531224 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 644 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.531275 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 644 election: Election decided. Result: candidate won.
I1014 19:27:34.531441 31283 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:34.531445 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 644.
I1014 19:27:34.531523 31283 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 FOLLOWER]: Leader election won for term 644
I1014 19:27:34.531603 31283 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.531745 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 644 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.531795 31283 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 506, Committed index: 506, Last appended: 642.506, Last appended by leader: 506, Current term: 644, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.532487 31802 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.532636 31802 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.532793 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:34.534206 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:34.535154 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:34.535802 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:34.537624 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 644 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 642 index: 506. Preceding OpId from leader: term: 644 index: 508. (index mismatch)
I1014 19:27:34.537704 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 642 index: 506. Preceding OpId from leader: term: 644 index: 508. (index mismatch)
I1014 19:27:34.538053 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 507, Last known committed idx: 506, Time since last communication: 0.000s
I1014 19:27:34.538231 31802 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 507, Last known committed idx: 506, Time since last communication: 0.000s
I1014 19:27:34.539618 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.539803 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.540153 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.540349 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.540962 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.541141 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.541280 31802 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.541463 31802 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.541673 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.541837 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.542451 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 644 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.542626 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.642403 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 644 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.642511 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:34.642573 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 644 FOLLOWER]: Advancing to term 645
I1014 19:27:34.644057 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.644536 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 645 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.644676 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 645 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.646775 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 645 candidate_status { last_received { term: 644 index: 508 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.646957 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 645 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.646944 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 645 candidate_status { last_received { term: 644 index: 508 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.647110 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 645 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.648031 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 645 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 645 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.648190 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 645 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 645 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.648245 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 645 election: Election decided. Result: candidate lost.
I1014 19:27:34.648627 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:34.648689 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Leader election lost for term 645. Reason: could not achieve majority
I1014 19:27:34.662700 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 644. Current term is 645. Ops: []
I1014 19:27:34.663332 31642 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 644.508, Next index: 509, Last known committed idx: 508, Time since last communication: 0.000s
I1014 19:27:34.663604 31642 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 LEADER]: Stepping down as leader of term 644
I1014 19:27:34.663655 31642 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.663839 31642 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 508, Committed index: 508, Last appended: 644.508, Last appended by leader: 508, Current term: 644, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.664014 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:34.664049 31642 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:34.664130 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 644 FOLLOWER]: Advancing to term 645
I1014 19:27:34.733705 30735 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:34.733781 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (starting election)
I1014 19:27:34.733831 30735 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 644 FOLLOWER]: Advancing to term 645
I1014 19:27:34.735159 30735 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.735558 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 645 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.735698 30735 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 645 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.736183 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 645 candidate_status { last_received { term: 644 index: 508 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:34.736222 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 645 candidate_status { last_received { term: 644 index: 508 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.736308 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:34.736395 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 645: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:34.736738 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 645 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 645: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:34.737860 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 645 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 645.
I1014 19:27:34.738140 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 645 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:34.738219 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 645 election: Election decided. Result: candidate won.
I1014 19:27:34.738392 30735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:34.738464 30735 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 FOLLOWER]: Leader election won for term 645
I1014 19:27:34.738732 30735 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:34.738930 30735 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 508, Committed index: 508, Last appended: 644.508, Last appended by leader: 508, Current term: 645, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.739675 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.739892 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.740314 31908 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:34.741686 31908 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:34.742621 31908 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:34.743217 31908 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:34.745082 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 645 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 644 index: 508. Preceding OpId from leader: term: 645 index: 510. (index mismatch)
I1014 19:27:34.745097 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 644 index: 508. Preceding OpId from leader: term: 645 index: 510. (index mismatch)
I1014 19:27:34.745502 31907 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 509, Last known committed idx: 508, Time since last communication: 0.000s
I1014 19:27:34.745719 30735 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 509, Last known committed idx: 508, Time since last communication: 0.000s
I1014 19:27:34.747133 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.747318 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.747335 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.747572 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.748332 25850 mvcc.cc:203] Tried to move safe_time back from 6306080582631288832 to 6306080582612574208. Current Snapshot: MvccSnapshot[committed={T|T < 6306080582631288832}]
I1014 19:27:34.748754 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.748920 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.749261 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.749447 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.750931 30735 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.751122 30735 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.752022 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 645 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.752161 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.948844 31642 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 645 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:34.948925 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:34.948972 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 645 FOLLOWER]: Advancing to term 646
I1014 19:27:34.951005 31642 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.951444 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 646 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:34.951616 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 646 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.952111 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 646 candidate_status { last_received { term: 645 index: 510 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:34.952296 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 645. Current term is 646. Ops: []
I1014 19:27:34.952229 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 646 candidate_status { last_received { term: 645 index: 510 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:34.952316 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 646 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.952378 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 645 FOLLOWER]: Advancing to term 646
I1014 19:27:34.952476 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.135s (vote granted)
I1014 19:27:34.952772 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 646 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 646 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:34.952910 31907 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 645.510, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I1014 19:27:34.953115 31909 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 LEADER]: Stepping down as leader of term 645
I1014 19:27:34.953155 31909 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:34.953316 31909 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 510, Committed index: 510, Last appended: 645.510, Last appended by leader: 510, Current term: 645, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.953529 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:34.953568 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:34.953678 31909 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 645 FOLLOWER]: Advancing to term 646
I1014 19:27:34.954550 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 646 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 646.
I1014 19:27:34.955098 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 646 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:34.955153 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 646 election: Election decided. Result: candidate won.
I1014 19:27:34.955535 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:34.955605 31642 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 FOLLOWER]: Leader election won for term 646
I1014 19:27:34.955720 31642 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:34.955915 31642 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 510, Committed index: 510, Last appended: 645.510, Last appended by leader: 510, Current term: 646, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:34.957183 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.957459 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.957751 31643 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:34.959812 31643 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:34.961117 31643 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:34.961836 31643 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:34.963624 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 646 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 645 index: 510. Preceding OpId from leader: term: 646 index: 512. (index mismatch)
I1014 19:27:34.963624 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 646 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 645 index: 510. Preceding OpId from leader: term: 646 index: 512. (index mismatch)
I1014 19:27:34.964107 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I1014 19:27:34.964478 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I1014 19:27:34.965618 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.965808 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.966012 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.966234 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.967525 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.967869 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.968541 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.968716 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:34.968578 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.968761 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:34.970556 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 646 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:34.970731 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.068081 31909 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 646 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:35.068161 31909 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:35.068212 31909 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 646 FOLLOWER]: Advancing to term 647
I1014 19:27:35.069684 31909 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.070055 31909 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 647 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.070194 31909 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 647 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.070654 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 647 candidate_status { last_received { term: 646 index: 512 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.070818 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 647 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.070742 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 647 candidate_status { last_received { term: 646 index: 512 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.070901 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 646 FOLLOWER]: Advancing to term 647
I1014 19:27:35.070971 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (vote granted)
I1014 19:27:35.071141 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 647 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 647 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.072551 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 647.
I1014 19:27:35.072870 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 647 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.072928 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 647 election: Election decided. Result: candidate won.
I1014 19:27:35.073101 31909 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:35.073153 31909 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 FOLLOWER]: Leader election won for term 647
I1014 19:27:35.073235 31909 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.073381 31909 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 512, Committed index: 512, Last appended: 646.512, Last appended by leader: 512, Current term: 647, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.074144 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.074321 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.074472 31908 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:35.075970 31908 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:35.077160 31908 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:35.078001 31908 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:35.080333 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 646 index: 512. Preceding OpId from leader: term: 647 index: 514. (index mismatch)
I1014 19:27:35.080368 28825 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 LEADER]: Stepping down as leader of term 646
I1014 19:27:35.080440 28825 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:35.080654 28825 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 512, Committed index: 512, Last appended: 646.512, Last appended by leader: 512, Current term: 646, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.080870 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:35.080914 28825 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:35.081050 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 646 FOLLOWER]: Advancing to term 647
I1014 19:27:35.081120 31909 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I1014 19:27:35.082895 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.083092 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.083297 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 647 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 646 index: 512. Preceding OpId from leader: term: 647 index: 514. (index mismatch)
I1014 19:27:35.084048 31910 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I1014 19:27:35.084512 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.084686 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.086163 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.086408 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.087525 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.087704 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.087803 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.087924 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.161170 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 647 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.161378 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.262279 31642 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 647 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.262348 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:35.262405 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 647 FOLLOWER]: Advancing to term 648
I1014 19:27:35.264410 31642 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 648 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.264878 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 648 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.265025 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 648 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.265588 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 648 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.265758 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 648 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.265714 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 648 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.265884 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 648 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.266168 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 648 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 648 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.266337 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 648 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 648 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.266396 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 648 election: Election decided. Result: candidate lost.
I1014 19:27:35.266578 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:35.266631 31642 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 648 FOLLOWER]: Leader election lost for term 648. Reason: could not achieve majority
I1014 19:27:35.275151 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 648 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 647. Current term is 648. Ops: []
I1014 19:27:35.275672 31907 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 647.514, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I1014 19:27:35.275883 31909 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 LEADER]: Stepping down as leader of term 647
I1014 19:27:35.275931 31909 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.276095 31909 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 514, Committed index: 514, Last appended: 647.514, Last appended by leader: 514, Current term: 647, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.276226 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:35.276262 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:35.276371 31909 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 647 FOLLOWER]: Advancing to term 648
I1014 19:27:35.353242 31909 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 648 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:35.353318 31909 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:35.353371 31909 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 648 FOLLOWER]: Advancing to term 649
I1014 19:27:35.354745 31909 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.355129 31909 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 649 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.355268 31909 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 649 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.355763 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 649 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.355924 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 648 FOLLOWER]: Advancing to term 649
I1014 19:27:35.355870 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 649 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.355999 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (vote granted)
I1014 19:27:35.356047 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 649 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.356379 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 649 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 649 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.357390 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 649.
I1014 19:27:35.357708 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 649 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.357718 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.357755 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 649 election: Election decided. Result: candidate won.
I1014 19:27:35.357761 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (starting election)
I1014 19:27:35.357807 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 647 FOLLOWER]: Advancing to term 648
I1014 19:27:35.357923 31909 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:35.357982 31909 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 FOLLOWER]: Leader election won for term 649
I1014 19:27:35.358070 31909 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.358232 31909 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 514, Committed index: 514, Last appended: 647.514, Last appended by leader: 514, Current term: 649, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.358944 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.359115 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.359295 31908 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:35.359383 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 648 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.359839 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 648 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.359984 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 648 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.360524 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 648 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.360682 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 648. Current term is 649.
W1014 19:27:35.361001 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 648 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 648. Current term is 649.
I1014 19:27:35.361035 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 648 election: Cancelling election due to peer responding with higher term
I1014 19:27:35.361070 31908 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:35.361217 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:35.361207 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 648 candidate_status { last_received { term: 647 index: 514 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.361280 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 648 FOLLOWER]: Advancing to term 649
I1014 19:27:35.361367 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 648 because replica is either leader or believes a valid leader to be alive.
W1014 19:27:35.361735 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 648 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 648 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.362433 31908 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:35.363322 31908 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:35.363343 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 649 FOLLOWER]: Leader election lost for term 648. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 648. Current term is 649.
I1014 19:27:35.365125 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 649 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 647 index: 514. Preceding OpId from leader: term: 649 index: 516. (index mismatch)
I1014 19:27:35.365141 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 647 index: 514. Preceding OpId from leader: term: 649 index: 516. (index mismatch)
I1014 19:27:35.365535 31909 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I1014 19:27:35.365689 31907 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 515, Last known committed idx: 514, Time since last communication: 0.000s
I1014 19:27:35.367082 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.367311 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.367302 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.367506 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.368253 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.368433 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.368520 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.368687 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.369251 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.369375 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.369319 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 649 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.369558 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.469971 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 649 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.470057 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:35.470116 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 649 FOLLOWER]: Advancing to term 650
I1014 19:27:35.472303 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.472770 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 650 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.472914 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 650 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.473493 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 650 candidate_status { last_received { term: 649 index: 516 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.473538 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 650 candidate_status { last_received { term: 649 index: 516 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.473664 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 650 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.473716 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 650 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.474442 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 650 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 650 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.474608 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 650 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 650 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.474666 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 650 election: Election decided. Result: candidate lost.
I1014 19:27:35.474943 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:35.475000 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Leader election lost for term 650. Reason: could not achieve majority
I1014 19:27:35.476434 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 649. Current term is 650. Ops: []
I1014 19:27:35.476902 31909 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 649.516, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I1014 19:27:35.477120 31909 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Stepping down as leader of term 649
I1014 19:27:35.477160 31909 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.477319 31909 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 649.516, Last appended by leader: 516, Current term: 649, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.477489 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:35.477555 31909 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:35.477661 31909 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 649 FOLLOWER]: Advancing to term 650
I1014 19:27:35.559144 31642 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.559226 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (starting election)
I1014 19:27:35.559278 31642 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 649 FOLLOWER]: Advancing to term 650
I1014 19:27:35.561308 31642 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.561800 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 650 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.561956 31642 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 650 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.562372 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 650 candidate_status { last_received { term: 649 index: 516 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.562557 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:35.562553 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 650 candidate_status { last_received { term: 649 index: 516 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.562736 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 650: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:35.563105 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 650 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 650: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:35.564080 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 650.
I1014 19:27:35.564342 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 650 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.564388 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 650 election: Election decided. Result: candidate won.
I1014 19:27:35.564604 31642 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.137s (election complete)
I1014 19:27:35.564658 31642 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 FOLLOWER]: Leader election won for term 650
I1014 19:27:35.564736 31642 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:35.564919 31642 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 649.516, Last appended by leader: 516, Current term: 650, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.566184 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.566311 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.566817 32222 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:35.568675 32222 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:35.569937 32222 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:35.570912 32222 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:35.573055 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 649 index: 516. Preceding OpId from leader: term: 650 index: 518. (index mismatch)
I1014 19:27:35.573137 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 649 index: 516. Preceding OpId from leader: term: 650 index: 518. (index mismatch)
I1014 19:27:35.573659 31283 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I1014 19:27:35.573812 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I1014 19:27:35.575367 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.575625 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.575799 31910 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.576108 31910 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.577961 31910 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.578140 31910 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.577461 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.578331 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.578608 31283 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.578910 31283 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.579944 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 650 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.580163 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.647984 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
I1014 19:27:35.680284 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:35.680375 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.103s (starting election)
I1014 19:27:35.680469 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 650 FOLLOWER]: Advancing to term 651
I1014 19:27:35.682626 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.683136 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 651 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.683553 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 651 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.683737 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 651 candidate_status { last_received { term: 650 index: 518 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.683907 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 651 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.684101 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 651 candidate_status { last_received { term: 650 index: 518 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.684198 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 651 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 651 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.684290 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 651 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.684654 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 651 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 651 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.684710 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 651 election: Election decided. Result: candidate lost.
I1014 19:27:35.685171 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:35.685230 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Leader election lost for term 651. Reason: could not achieve majority
I1014 19:27:35.685567 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 650. Current term is 651. Ops: []
I1014 19:27:35.686056 32224 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 650.518, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I1014 19:27:35.686303 32224 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 LEADER]: Stepping down as leader of term 650
I1014 19:27:35.686354 32224 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:35.686558 32224 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 650.518, Last appended by leader: 518, Current term: 650, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.686774 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:35.686817 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:35.686945 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 650 FOLLOWER]: Advancing to term 651
I1014 19:27:35.757892 31910 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:35.757972 31910 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:35.758024 31910 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 650 FOLLOWER]: Advancing to term 651
I1014 19:27:35.759620 31910 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.759984 31910 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 651 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.760112 31910 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 651 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.760633 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 651 candidate_status { last_received { term: 650 index: 518 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.760669 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 651 candidate_status { last_received { term: 650 index: 518 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.760776 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (vote granted)
I1014 19:27:35.760821 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 651: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:35.761137 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 651 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 651: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:35.762246 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 651 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 651.
I1014 19:27:35.762522 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 651 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.762563 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 651 election: Election decided. Result: candidate won.
I1014 19:27:35.762724 31910 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (election complete)
I1014 19:27:35.762787 31910 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 FOLLOWER]: Leader election won for term 651
I1014 19:27:35.762914 31910 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.763123 31910 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 650.518, Last appended by leader: 518, Current term: 651, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.764256 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.764436 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.764626 31908 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:35.766116 31908 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:35.767081 31908 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:35.767776 31908 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:35.769539 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 650 index: 518. Preceding OpId from leader: term: 651 index: 520. (index mismatch)
I1014 19:27:35.769630 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 651 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 650 index: 518. Preceding OpId from leader: term: 651 index: 520. (index mismatch)
I1014 19:27:35.769989 31910 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I1014 19:27:35.770195 31909 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I1014 19:27:35.771711 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.771906 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.772084 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.772275 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.772828 31909 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.773016 31909 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.772975 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.773134 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.773871 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.773807 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 651 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.774065 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.774075 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.874215 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 651 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.874291 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:35.874334 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 651 FOLLOWER]: Advancing to term 652
I1014 19:27:35.876744 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.877243 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 652 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.877416 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 652 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:35.877951 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 652 candidate_status { last_received { term: 651 index: 520 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.878129 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 652 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.878176 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 652 candidate_status { last_received { term: 651 index: 520 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:35.878410 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 652 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.878604 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 652 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 652 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.878844 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 652 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 652 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:35.878916 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 652 election: Election decided. Result: candidate lost.
I1014 19:27:35.879204 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:35.879267 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Leader election lost for term 652. Reason: could not achieve majority
I1014 19:27:35.889252 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 651. Current term is 652. Ops: []
I1014 19:27:35.890099 31910 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 651.520, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I1014 19:27:35.890336 31907 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 LEADER]: Stepping down as leader of term 651
I1014 19:27:35.890377 31907 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:35.890611 31907 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 520, Committed index: 520, Last appended: 651.520, Last appended by leader: 520, Current term: 651, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.890832 31907 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:35.890873 31907 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:35.890969 31907 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 651 FOLLOWER]: Advancing to term 652
I1014 19:27:35.960650 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:35.960745 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (starting election)
I1014 19:27:35.960820 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 651 FOLLOWER]: Advancing to term 652
I1014 19:27:35.962342 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.962731 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 652 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.962899 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 652 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:35.963371 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 652 candidate_status { last_received { term: 651 index: 520 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:35.963485 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 652 candidate_status { last_received { term: 651 index: 520 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:35.963536 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (vote granted)
I1014 19:27:35.963610 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 652: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:35.963948 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 652 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 652: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:35.965083 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 652 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 652.
I1014 19:27:35.965386 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 652 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:35.965451 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 652 election: Election decided. Result: candidate won.
I1014 19:27:35.965688 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:35.965786 31540 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 FOLLOWER]: Leader election won for term 652
I1014 19:27:35.966035 31540 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:35.966238 31540 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 520, Committed index: 520, Last appended: 651.520, Last appended by leader: 520, Current term: 652, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:35.967147 32344 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.967384 32344 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.967869 32345 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:35.969403 32345 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:35.970438 32345 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:35.971673 32345 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:35.973489 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 652 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 651 index: 520. Preceding OpId from leader: term: 652 index: 522. (index mismatch)
I1014 19:27:35.973502 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 651 index: 520. Preceding OpId from leader: term: 652 index: 522. (index mismatch)
I1014 19:27:35.973983 32344 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I1014 19:27:35.974242 31540 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 521, Last known committed idx: 520, Time since last communication: 0.000s
I1014 19:27:35.975781 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.976028 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.975829 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.976081 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.977632 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.977841 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.978071 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.978255 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:35.978633 32344 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.978960 32344 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:35.979398 32346 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 652 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:35.979671 32346 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.156572 31907 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 652 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:36.156653 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:36.156705 31907 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 652 FOLLOWER]: Advancing to term 653
I1014 19:27:36.158085 31907 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.158547 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 653 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:36.158691 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 653 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.159293 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 653 candidate_status { last_received { term: 652 index: 522 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:36.159284 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 653 candidate_status { last_received { term: 652 index: 522 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:36.159505 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 653 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.159519 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 653 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.159864 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 653 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 653 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.159967 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 653 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 653 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.160027 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 653 election: Election decided. Result: candidate lost.
I1014 19:27:36.160432 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.111s (election complete)
I1014 19:27:36.160511 31907 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Leader election lost for term 653. Reason: could not achieve majority
I1014 19:27:36.160773 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 652. Current term is 653. Ops: []
I1014 19:27:36.161486 32346 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 652.522, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I1014 19:27:36.161736 32347 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 LEADER]: Stepping down as leader of term 652
I1014 19:27:36.161810 32347 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:36.162073 32347 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 522, Committed index: 522, Last appended: 652.522, Last appended by leader: 522, Current term: 652, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.162336 32347 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:36.162389 32347 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:36.162549 32347 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 652 FOLLOWER]: Advancing to term 653
I1014 19:27:36.166523 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:36.166587 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:36.166630 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 652 FOLLOWER]: Advancing to term 653
I1014 19:27:36.176872 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.177754 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 653 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.177897 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 653 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.178282 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 653 candidate_status { last_received { term: 652 index: 522 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:36.178397 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 653 candidate_status { last_received { term: 652 index: 522 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:36.178463 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 653: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:36.178563 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (vote granted)
I1014 19:27:36.178761 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 653 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 653: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:36.180065 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 653 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 653.
I1014 19:27:36.180336 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 653 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.180385 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 653 election: Election decided. Result: candidate won.
I1014 19:27:36.180572 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (election complete)
I1014 19:27:36.180660 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 FOLLOWER]: Leader election won for term 653
I1014 19:27:36.180747 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:36.180922 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 522, Committed index: 522, Last appended: 652.522, Last appended by leader: 522, Current term: 653, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.181895 31642 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.182014 31642 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.183037 32433 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:36.184358 32433 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:36.185309 32433 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:36.185967 32433 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:36.187701 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 652 index: 522. Preceding OpId from leader: term: 653 index: 524. (index mismatch)
I1014 19:27:36.187767 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 653 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 652 index: 522. Preceding OpId from leader: term: 653 index: 524. (index mismatch)
I1014 19:27:36.188091 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I1014 19:27:36.188329 31642 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I1014 19:27:36.189678 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.189868 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.189944 32347 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.190141 32347 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.191751 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.191915 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.193238 32347 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.193590 32347 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.196866 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.197038 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.197760 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 653 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.198026 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.293042 32347 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 653 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:36.293153 32347 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.147s (starting election)
I1014 19:27:36.293226 32347 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 653 FOLLOWER]: Advancing to term 654
I1014 19:27:36.295773 32347 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.296499 32347 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 654 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.296927 32347 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 654 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:36.297453 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 654 candidate_status { last_received { term: 653 index: 524 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:36.297531 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 654 candidate_status { last_received { term: 653 index: 524 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:36.297637 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 653 FOLLOWER]: Advancing to term 654
I1014 19:27:36.297715 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 654 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.297708 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.103s (vote granted)
I1014 19:27:36.298230 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 654 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 654 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.299753 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 654.
I1014 19:27:36.300859 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 654 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.300930 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 654 election: Election decided. Result: candidate won.
I1014 19:27:36.301308 32347 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:36.301450 32347 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 FOLLOWER]: Leader election won for term 654
I1014 19:27:36.301692 32347 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:36.301959 32347 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 524, Committed index: 524, Last appended: 653.524, Last appended by leader: 524, Current term: 654, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.303843 32346 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.304034 32346 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.304270 32345 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:36.306462 32345 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:36.306946 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 653. Current term is 654. Ops: []
I1014 19:27:36.307572 32434 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 653.524, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I1014 19:27:36.307796 32434 consensus_queue.cc:1104] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer 4043e04130264237805f012d8e251bd4 log is divergent from this leader: its last log entry 654.525 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 524
I1014 19:27:36.307880 32434 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 LEADER]: Stepping down as leader of term 653
I1014 19:27:36.307915 32434 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:36.307977 32345 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:36.308128 32434 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 524, Committed index: 524, Last appended: 653.524, Last appended by leader: 524, Current term: 653, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.308305 32434 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:36.308346 32434 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:36.308504 32434 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 653 FOLLOWER]: Advancing to term 654
I1014 19:27:36.308943 32345 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:36.311377 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 654 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 653 index: 524. Preceding OpId from leader: term: 654 index: 526. (index mismatch)
I1014 19:27:36.311847 32347 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I1014 19:27:36.312551 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 653 index: 524. Preceding OpId from leader: term: 654 index: 526. (index mismatch)
I1014 19:27:36.312984 32347 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I1014 19:27:36.313247 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.313457 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.314435 32346 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.314796 32346 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.315186 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.315376 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.315259 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.315449 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.316370 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.316630 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.317277 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 654 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.317435 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.414814 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 654 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:36.414891 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:36.414947 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 654 FOLLOWER]: Advancing to term 655
I1014 19:27:36.416471 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.416949 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 655 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.417098 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 655 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.417543 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 655 candidate_status { last_received { term: 654 index: 526 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:36.417714 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 655 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.417816 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 655 candidate_status { last_received { term: 654 index: 526 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:36.418030 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 655 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.418113 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 655 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 655 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.418609 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 655 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 655 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.418665 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 655 election: Election decided. Result: candidate lost.
I1014 19:27:36.419623 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:36.419684 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Leader election lost for term 655. Reason: could not achieve majority
I1014 19:27:36.420100 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 654. Current term is 655. Ops: []
I1014 19:27:36.420562 32344 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 654.526, Next index: 527, Last known committed idx: 526, Time since last communication: 0.000s
I1014 19:27:36.420960 31540 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Stepping down as leader of term 654
I1014 19:27:36.421017 31540 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:36.421216 31540 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 526, Committed index: 526, Last appended: 654.526, Last appended by leader: 526, Current term: 654, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.421442 31540 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:36.421509 31540 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:36.421636 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 654 FOLLOWER]: Advancing to term 655
I1014 19:27:36.495968 31907 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:36.496039 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:36.496074 31907 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 654 FOLLOWER]: Advancing to term 655
I1014 19:27:36.498136 31907 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.498666 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 655 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:36.498821 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 655 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.499447 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 655 candidate_status { last_received { term: 654 index: 526 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:36.499583 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 655 candidate_status { last_received { term: 654 index: 526 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:36.499667 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (vote granted)
I1014 19:27:36.499756 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 655: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:36.500161 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 655 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 655: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:36.501868 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 655.
I1014 19:27:36.502277 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 655 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.502333 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 655 election: Election decided. Result: candidate won.
I1014 19:27:36.502652 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:36.502704 31907 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 FOLLOWER]: Leader election won for term 655
I1014 19:27:36.502967 31907 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:36.503165 31907 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 526, Committed index: 526, Last appended: 654.526, Last appended by leader: 526, Current term: 655, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.504034 32525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.504235 32525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.504632 32526 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:36.506026 32526 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:36.506980 32526 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:36.507606 32526 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:36.509505 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 654 index: 526. Preceding OpId from leader: term: 655 index: 528. (index mismatch)
I1014 19:27:36.509541 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 654 index: 526. Preceding OpId from leader: term: 655 index: 528. (index mismatch)
I1014 19:27:36.509929 32525 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 527, Last known committed idx: 526, Time since last communication: 0.000s
I1014 19:27:36.510090 31907 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 527, Last known committed idx: 526, Time since last communication: 0.000s
I1014 19:27:36.511517 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.511855 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.512092 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.512284 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.512820 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.512935 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.513062 32525 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.513348 32525 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.513703 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.513965 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.514621 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 655 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.514900 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.778261 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:36.778342 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (starting election)
I1014 19:27:36.778396 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 655 FOLLOWER]: Advancing to term 656
I1014 19:27:36.780534 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.780982 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 656 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.781118 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 656 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:36.781702 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 656 candidate_status { last_received { term: 655 index: 528 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:36.781893 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 656 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.782263 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 656 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 656 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.782282 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 656 candidate_status { last_received { term: 655 index: 528 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:36.782481 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 656 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.782953 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 656 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 656 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:36.783017 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 656 election: Election decided. Result: candidate lost.
I1014 19:27:36.783344 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:36.783463 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Leader election lost for term 656. Reason: could not achieve majority
I1014 19:27:36.797194 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 655. Current term is 656. Ops: []
I1014 19:27:36.797722 32527 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 655.528, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I1014 19:27:36.797933 32527 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 LEADER]: Stepping down as leader of term 655
I1014 19:27:36.797967 32527 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:36.798135 32527 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 655.528, Last appended by leader: 528, Current term: 655, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.798326 32527 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:36.798367 32527 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:36.798470 32527 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 655 FOLLOWER]: Advancing to term 656
I1014 19:27:36.884587 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:36.884675 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (starting election)
I1014 19:27:36.884726 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 655 FOLLOWER]: Advancing to term 656
I1014 19:27:36.887043 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.887539 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 656 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.887708 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 656 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:36.888284 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 656 candidate_status { last_received { term: 655 index: 528 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:36.888334 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 656 candidate_status { last_received { term: 655 index: 528 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:36.888425 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (vote granted)
I1014 19:27:36.888545 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 656: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:36.888936 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 656 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 656: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:36.889886 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 656 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 656.
I1014 19:27:36.890167 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 656 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:36.890218 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 656 election: Election decided. Result: candidate won.
I1014 19:27:36.890430 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:36.890518 31540 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 FOLLOWER]: Leader election won for term 656
I1014 19:27:36.890606 31540 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:36.890776 31540 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 655.528, Last appended by leader: 528, Current term: 656, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:36.891613 32344 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.891750 32344 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.892135 32695 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:36.894994 32695 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:36.896519 32695 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:36.897472 32695 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:36.899292 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 656 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 655 index: 528. Preceding OpId from leader: term: 656 index: 530. (index mismatch)
I1014 19:27:36.899351 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 655 index: 528. Preceding OpId from leader: term: 656 index: 530. (index mismatch)
I1014 19:27:36.899760 31540 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I1014 19:27:36.899988 32344 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I1014 19:27:36.901453 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.901669 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.901787 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.901978 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.903054 32344 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.903239 32344 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.903252 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.903462 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:36.903975 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.904145 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:36.904381 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 656 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:36.904577 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.004864 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:37.004945 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:37.004998 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 656 FOLLOWER]: Advancing to term 657
I1014 19:27:37.006635 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.006906 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 656. Current term is 657. Ops: []
I1014 19:27:37.007073 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 657 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.007203 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 657 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.007632 32696 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 656.530, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I1014 19:27:37.007905 31540 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 LEADER]: Stepping down as leader of term 656
I1014 19:27:37.007946 31540 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:37.008002 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 657 candidate_status { last_received { term: 656 index: 530 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.008119 31540 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 530, Committed index: 530, Last appended: 656.530, Last appended by leader: 530, Current term: 656, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.008293 31540 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:37.008337 31540 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:37.008328 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 657 candidate_status { last_received { term: 656 index: 530 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.008500 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 656 FOLLOWER]: Advancing to term 657
I1014 19:27:37.008525 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 656 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 657 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.008976 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 657 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 656 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 657 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.010149 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:37.011620 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 657 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 657.
I1014 19:27:37.011941 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 657 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.012006 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 657 election: Election decided. Result: candidate won.
I1014 19:27:37.012176 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:37.012233 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 FOLLOWER]: Leader election won for term 657
I1014 19:27:37.012544 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:37.012737 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 530, Committed index: 530, Last appended: 656.530, Last appended by leader: 530, Current term: 657, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.013732 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.013945 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.018831 32748 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:37.020949 32748 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:37.023743 32748 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:37.024574 32748 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:37.026933 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 657 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 656 index: 530. Preceding OpId from leader: term: 657 index: 532. (index mismatch)
I1014 19:27:37.027308 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 656 FOLLOWER]: Advancing to term 657
I1014 19:27:37.027453 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I1014 19:27:37.028980 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.029170 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.029367 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 657 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 656 index: 530. Preceding OpId from leader: term: 657 index: 532. (index mismatch)
I1014 19:27:37.029852 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 531, Last known committed idx: 530, Time since last communication: 0.000s
I1014 19:27:37.032703 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.032860 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.034281 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.034462 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.035495 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.035763 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.036459 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.036677 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.044229 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 657 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.044540 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.135555 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 657 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:37.135634 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:37.135684 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 657 FOLLOWER]: Advancing to term 658
I1014 19:27:37.137904 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.138411 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 658 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.138785 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 658 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.138999 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 658 candidate_status { last_received { term: 657 index: 532 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.139185 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 657 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 658 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.139355 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 658 candidate_status { last_received { term: 657 index: 532 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.139559 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 658 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.139832 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 658 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 657 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 658 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.139976 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 658 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 658 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.140028 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 658 election: Election decided. Result: candidate lost.
I1014 19:27:37.140200 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:37.140280 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Leader election lost for term 658. Reason: could not achieve majority
I1014 19:27:37.158218 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 657. Current term is 658. Ops: []
I1014 19:27:37.158677 32750 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 657.532, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I1014 19:27:37.158921 32747 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 LEADER]: Stepping down as leader of term 657
I1014 19:27:37.158972 32747 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:37.159164 32747 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 532, Committed index: 532, Last appended: 657.532, Last appended by leader: 532, Current term: 657, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.159301 32747 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:37.159333 32747 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:37.159421 32747 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 657 FOLLOWER]: Advancing to term 658
I1014 19:27:37.227084 31907 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 657 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:37.227176 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:37.227233 31907 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 657 FOLLOWER]: Advancing to term 658
I1014 19:27:37.228757 31907 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.229138 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 658 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.229274 31907 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 658 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.229919 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 658 candidate_status { last_received { term: 657 index: 532 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.229919 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 658 candidate_status { last_received { term: 657 index: 532 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.230083 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:37.230113 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 658: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.230530 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 658 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 658: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.231549 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 658.
I1014 19:27:37.231995 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 658 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.232051 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 658 election: Election decided. Result: candidate won.
I1014 19:27:37.232364 31907 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.113s (election complete)
I1014 19:27:37.232424 31907 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 FOLLOWER]: Leader election won for term 658
I1014 19:27:37.232537 31907 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:37.232673 31907 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 532, Committed index: 532, Last appended: 657.532, Last appended by leader: 532, Current term: 658, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.233386 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.233541 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.233973   348 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:37.235435   348 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:37.236376   348 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:37.236994   348 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:37.238823 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 657 index: 532. Preceding OpId from leader: term: 658 index: 534. (index mismatch)
I1014 19:27:37.238822 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 657 index: 532. Preceding OpId from leader: term: 658 index: 534. (index mismatch)
I1014 19:27:37.239276 31907 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I1014 19:27:37.239415 32527 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I1014 19:27:37.240737 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.240923 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.241128 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.241312 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.242708 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.242873 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.242980 31907 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.243181 31907 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.243882 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.244051 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.244644 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 658 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.244834 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.344887 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:37.344961 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:37.345011 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 658 FOLLOWER]: Advancing to term 659
I1014 19:27:37.347159 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.347564 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 659 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.347689 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 659 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.348219 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 659 candidate_status { last_received { term: 658 index: 534 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.348217 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 659 candidate_status { last_received { term: 658 index: 534 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.348369 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 659 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.348369 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 659 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.348703 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 659 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 659 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.348909 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 659 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 659 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.348966 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 659 election: Election decided. Result: candidate lost.
I1014 19:27:37.349208 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:37.349288 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Leader election lost for term 659. Reason: could not achieve majority
I1014 19:27:37.356474 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 658. Current term is 659. Ops: []
I1014 19:27:37.357101 31907 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 658.534, Next index: 535, Last known committed idx: 534, Time since last communication: 0.000s
I1014 19:27:37.357326 31907 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 LEADER]: Stepping down as leader of term 658
I1014 19:27:37.357364 31907 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:37.357560 31907 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 534, Committed index: 534, Last appended: 658.534, Last appended by leader: 534, Current term: 658, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.357729 31907 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:37.357767 31907 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:37.357856 31907 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 658 FOLLOWER]: Advancing to term 659
I1014 19:27:37.419562 32747 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:37.419641 32747 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:37.419713 32747 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 658 FOLLOWER]: Advancing to term 659
I1014 19:27:37.421151 32747 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.421555 32747 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 659 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.421696 32747 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 659 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.422276 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 659 candidate_status { last_received { term: 658 index: 534 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.422276 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 659 candidate_status { last_received { term: 658 index: 534 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.422423 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (vote granted)
I1014 19:27:37.422448 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 659: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.422833 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 659 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 659: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.423988 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 659 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 659.
I1014 19:27:37.424298 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 659 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.424345 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 659 election: Election decided. Result: candidate won.
I1014 19:27:37.424540 32747 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.119s (election complete)
I1014 19:27:37.424595 32747 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 FOLLOWER]: Leader election won for term 659
I1014 19:27:37.424676 32747 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:37.424839 32747 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 534, Committed index: 534, Last appended: 658.534, Last appended by leader: 534, Current term: 659, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.425839 32750 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.426093 32750 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.426277 32748 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:37.427983 32748 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:37.429162 32748 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:37.429927 32748 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:37.431936 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 658 index: 534. Preceding OpId from leader: term: 659 index: 536. (index mismatch)
I1014 19:27:37.432040 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 659 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 658 index: 534. Preceding OpId from leader: term: 659 index: 536. (index mismatch)
I1014 19:27:37.432472 32750 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 535, Last known committed idx: 534, Time since last communication: 0.000s
I1014 19:27:37.432677 32747 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 535, Last known committed idx: 534, Time since last communication: 0.000s
I1014 19:27:37.434043 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.434228 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.434264 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.434463 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.435346 32750 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.435560 32750 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.435518 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.435685 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.435938 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.436113 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.436586 25847 mvcc.cc:203] Tried to move safe_time back from 6306080593636552704 to 6306080593614024704. Current Snapshot: MvccSnapshot[committed={T|T < 6306080593636552704 or (T in {6306080593636552704})}]
I1014 19:27:37.436556 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 659 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.436733 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.535805 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:37.535904 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.108s (starting election)
I1014 19:27:37.535944 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 659 FOLLOWER]: Advancing to term 660
I1014 19:27:37.537158 32527 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 659 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:37.537225 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:37.537274 32527 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 659 FOLLOWER]: Advancing to term 660
I1014 19:27:37.537696 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.538187 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 660 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.538352 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 660 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.538986 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 660 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.539170 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 660 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.539120 32527 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.539497 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 660 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.539520 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 660 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.539611 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 660 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 660 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.539641 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 660 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.539685 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 660: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:37.540177 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 660 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.540314 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 660 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.540469 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 660 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.540632 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 660: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.540714 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 660 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 660 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.541133 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 660 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 660: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:37.541204 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 660 election: Election decided. Result: candidate lost.
I1014 19:27:37.541528 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 660 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 660: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:37.541586 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 660 election: Election decided. Result: candidate lost.
I1014 19:27:37.541838 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:37.541888 32527 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Leader election lost for term 660. Reason: could not achieve majority
I1014 19:27:37.542131 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:37.542182 31540 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Leader election lost for term 660. Reason: could not achieve majority
I1014 19:27:37.551780 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 659. Current term is 660. Ops: []
I1014 19:27:37.552462 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 659. Current term is 660. Ops: []
I1014 19:27:37.552759 32747 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 659.536, Next index: 537, Last known committed idx: 536, Time since last communication: 0.000s
I1014 19:27:37.552995 32224 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 659.536, Next index: 537, Last known committed idx: 536, Time since last communication: 0.000s
I1014 19:27:37.553200 32747 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Stepping down as leader of term 659
I1014 19:27:37.553244 32747 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:37.553400 32747 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 536, Committed index: 536, Last appended: 659.536, Last appended by leader: 536, Current term: 659, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.553551 32747 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:37.553576 32747 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:37.553650 32747 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 659 FOLLOWER]: Advancing to term 660
I1014 19:27:37.643456 31540 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:37.643538 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.200s (starting election)
I1014 19:27:37.643594 31540 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 660 FOLLOWER]: Advancing to term 661
I1014 19:27:37.645184 31540 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.645632 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 661 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.645792 31540 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 661 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.646466 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 661 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.646445 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 661 candidate_status { last_received { term: 659 index: 536 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.646617 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 660 FOLLOWER]: Advancing to term 661
I1014 19:27:37.646617 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 660 FOLLOWER]: Advancing to term 661
I1014 19:27:37.646687 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.189s (vote granted)
I1014 19:27:37.646687 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.110s (vote granted)
I1014 19:27:37.648769 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 661 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 661.
I1014 19:27:37.648793 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 661.
I1014 19:27:37.649243 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 661 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.649317 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 661 election: Election decided. Result: candidate won.
I1014 19:27:37.649559 31540 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:37.649585 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 661 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.649627 31540 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 FOLLOWER]: Leader election won for term 661
I1014 19:27:37.649948 31540 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:37.650200 31540 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 536, Committed index: 536, Last appended: 659.536, Last appended by leader: 536, Current term: 661, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.651135   467 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.651523   467 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.652015   468 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:37.655113   468 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:37.655131 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:37.656636   468 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:37.657392   468 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:37.659570 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 659 index: 536. Preceding OpId from leader: term: 661 index: 538. (index mismatch)
I1014 19:27:37.659818 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 661 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 659 index: 536. Preceding OpId from leader: term: 661 index: 538. (index mismatch)
I1014 19:27:37.660225 31540 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 537, Last known committed idx: 536, Time since last communication: 0.000s
I1014 19:27:37.660418   467 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 537, Last known committed idx: 536, Time since last communication: 0.000s
I1014 19:27:37.662045 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.662061 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.662235 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.662248 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.663309 31540 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.663529 31540 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.665356 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.665539 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.665922   467 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.666299   467 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.666887 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 661 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.667079 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.850611 32527 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 661 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:37.850683 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (starting election)
I1014 19:27:37.850725 32527 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 661 FOLLOWER]: Advancing to term 662
I1014 19:27:37.852886 32527 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.853343 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 662 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:37.853502 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 662 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.854053 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 662 candidate_status { last_received { term: 661 index: 538 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.854239 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 662 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.854338 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 662 candidate_status { last_received { term: 661 index: 538 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:37.854518 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 662 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.854676 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 662 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 662 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.854859 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 662 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 662 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:37.854921 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 662 election: Election decided. Result: candidate lost.
I1014 19:27:37.855554 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (election complete)
I1014 19:27:37.855615 32527 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Leader election lost for term 662. Reason: could not achieve majority
I1014 19:27:37.859908 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 661. Current term is 662. Ops: []
I1014 19:27:37.860632   467 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 661.538, Next index: 539, Last known committed idx: 538, Time since last communication: 0.000s
I1014 19:27:37.860888   467 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 LEADER]: Stepping down as leader of term 661
I1014 19:27:37.860929   467 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:37.861114   467 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 538, Committed index: 538, Last appended: 661.538, Last appended by leader: 538, Current term: 661, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.861250   467 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:37.861290   467 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:37.861400   467 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 661 FOLLOWER]: Advancing to term 662
I1014 19:27:37.866811 32747 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:37.866895 32747 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:37.866948 32747 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 661 FOLLOWER]: Advancing to term 662
I1014 19:27:37.869026 32747 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.869541 32747 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 662 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:37.869704 32747 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 662 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.870298 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 662 candidate_status { last_received { term: 661 index: 538 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:37.870512 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 662: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:37.870812 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 662 candidate_status { last_received { term: 661 index: 538 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:37.870986 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 662 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 662: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:37.871018 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (vote granted)
I1014 19:27:37.873297 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 662 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 662.
I1014 19:27:37.874316 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 662 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:37.874375 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 662 election: Election decided. Result: candidate won.
I1014 19:27:37.875579 32747 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:37.875659 32747 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 FOLLOWER]: Leader election won for term 662
I1014 19:27:37.875777 32747 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:37.876000 32747 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 538, Committed index: 538, Last appended: 661.538, Last appended by leader: 538, Current term: 662, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:37.877058 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.877267 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.877483 32748 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:37.879662 32748 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:37.880940 32748 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:37.881871 32748 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:37.884030 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 661 index: 538. Preceding OpId from leader: term: 662 index: 540. (index mismatch)
I1014 19:27:37.884565 32747 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 539, Last known committed idx: 538, Time since last communication: 0.000s
I1014 19:27:37.885273 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 662 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 661 index: 538. Preceding OpId from leader: term: 662 index: 540. (index mismatch)
I1014 19:27:37.885876 32747 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 539, Last known committed idx: 538, Time since last communication: 0.000s
I1014 19:27:37.887140 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.887372 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.888413   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.888695   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.889554 32747 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.889730 32747 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.889598 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.889768 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:37.890607 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.890781 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:37.890934   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 662 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:37.891119   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.068794   469 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 662 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:38.068904   469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:38.068959   469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 662 FOLLOWER]: Advancing to term 663
I1014 19:27:38.071071   469 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.071581   469 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 663 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.071722   469 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 663 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.072259 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 663 candidate_status { last_received { term: 662 index: 540 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.072316 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 663 candidate_status { last_received { term: 662 index: 540 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:38.072508 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 663 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.072511 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 663 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.072903 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 663 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 663 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.073032 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 663 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 663 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.073097 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 663 election: Election decided. Result: candidate lost.
I1014 19:27:38.073266   469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:38.073352   469 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Leader election lost for term 663. Reason: could not achieve majority
I1014 19:27:38.081334 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 662. Current term is 663. Ops: []
I1014 19:27:38.081926 32750 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 662.540, Next index: 541, Last known committed idx: 540, Time since last communication: 0.000s
I1014 19:27:38.082157 32224 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 LEADER]: Stepping down as leader of term 662
I1014 19:27:38.082203 32224 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:38.082376 32224 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 540, Committed index: 540, Last appended: 662.540, Last appended by leader: 540, Current term: 662, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.082561 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:38.082602 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:38.082725 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 662 FOLLOWER]: Advancing to term 663
I1014 19:27:38.090350 32527 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:38.090416 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (starting election)
I1014 19:27:38.090462 32527 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 662 FOLLOWER]: Advancing to term 663
I1014 19:27:38.092511 32527 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.093025 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 663 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.093174 32527 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 663 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.093680 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 663 candidate_status { last_received { term: 662 index: 540 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:38.093719 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 663 candidate_status { last_received { term: 662 index: 540 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.093869 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 663: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:38.093858 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:38.094228 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 663 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 663: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:38.095340 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 663 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 663.
I1014 19:27:38.095667 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 663 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.095713 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 663 election: Election decided. Result: candidate won.
I1014 19:27:38.095903 32527 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:38.095959 32527 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 FOLLOWER]: Leader election won for term 663
I1014 19:27:38.096249 32527 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:38.096482 32527 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 540, Committed index: 540, Last appended: 662.540, Last appended by leader: 540, Current term: 663, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.097337   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.097647   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.098107   666 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:38.099537   666 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:38.100425   666 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:38.101078   666 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:38.102861 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 663 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 662 index: 540. Preceding OpId from leader: term: 663 index: 542. (index mismatch)
I1014 19:27:38.102867 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 662 index: 540. Preceding OpId from leader: term: 663 index: 542. (index mismatch)
I1014 19:27:38.103338 32527 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 541, Last known committed idx: 540, Time since last communication: 0.000s
I1014 19:27:38.103502   665 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 541, Last known committed idx: 540, Time since last communication: 0.000s
I1014 19:27:38.104954 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.104954   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.105146 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.105146   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.107355   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.107615   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.107774   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.107961   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.107416 32527 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.108147 32527 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.108158 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 663 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.108331 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.395773 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 663 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:38.395853 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:38.395902 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 663 FOLLOWER]: Advancing to term 664
I1014 19:27:38.397390 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.397752 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 664 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.397905 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 664 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.398363 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 664 candidate_status { last_received { term: 663 index: 542 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:38.398581 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 664 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.398859 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 664 candidate_status { last_received { term: 663 index: 542 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:38.398969 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 664 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 664 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.399036 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 664 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.399472 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 664 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 664 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.399550 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 664 election: Election decided. Result: candidate lost.
I1014 19:27:38.399716 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:38.399770 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Leader election lost for term 664. Reason: could not achieve majority
I1014 19:27:38.399919 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 663. Current term is 664. Ops: []
I1014 19:27:38.400333   667 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 663.542, Next index: 543, Last known committed idx: 542, Time since last communication: 0.000s
I1014 19:27:38.400617   667 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 LEADER]: Stepping down as leader of term 663
I1014 19:27:38.400661   667 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:38.400808   667 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 542, Committed index: 542, Last appended: 663.542, Last appended by leader: 542, Current term: 663, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.400992   667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:38.401024   667 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:38.401124   667 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 663 FOLLOWER]: Advancing to term 664
I1014 19:27:38.469393   469 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:38.469496   469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:38.469553   469 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 663 FOLLOWER]: Advancing to term 664
I1014 19:27:38.471019   469 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.471457   469 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 664 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.471601   469 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 664 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.472090 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 664 candidate_status { last_received { term: 663 index: 542 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:38.472112 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 664 candidate_status { last_received { term: 663 index: 542 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.472219 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.108s (vote granted)
I1014 19:27:38.472254 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 664: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:38.472565 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 664 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 664: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:38.473613 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 664 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 664.
I1014 19:27:38.473922 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 664 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.473966 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 664 election: Election decided. Result: candidate won.
I1014 19:27:38.474114   469 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:38.474179   469 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 FOLLOWER]: Leader election won for term 664
I1014 19:27:38.474472   469 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:38.474632   469 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 542, Committed index: 542, Last appended: 663.542, Last appended by leader: 542, Current term: 664, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.475339   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.475586   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.475950   840 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:38.477993   840 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:38.479179   840 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:38.479866   840 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:38.481624 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 664 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 663 index: 542. Preceding OpId from leader: term: 664 index: 544. (index mismatch)
I1014 19:27:38.481631 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 663 index: 542. Preceding OpId from leader: term: 664 index: 544. (index mismatch)
I1014 19:27:38.482025   839 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 543, Last known committed idx: 542, Time since last communication: 0.000s
I1014 19:27:38.482147   469 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 543, Last known committed idx: 542, Time since last communication: 0.000s
I1014 19:27:38.483630 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.483845 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.483703   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.484055   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.485468 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.485707 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.485828   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.486008   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.485103   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.486219   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.486385   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 664 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.486634   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.586236   665 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 664 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:38.586329   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:38.586390   665 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 664 FOLLOWER]: Advancing to term 665
I1014 19:27:38.588546   665 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.589033   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 665 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.589181   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 665 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.589895 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 665 candidate_status { last_received { term: 664 index: 544 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:38.590095 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 665 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.590567 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 665 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 665 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.590809 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 665 candidate_status { last_received { term: 664 index: 544 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.591020 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 665 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.591369 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 665 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 665 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.591469 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 665 election: Election decided. Result: candidate lost.
I1014 19:27:38.591678   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:38.591739   665 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Leader election lost for term 665. Reason: could not achieve majority
I1014 19:27:38.599301 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 664. Current term is 665. Ops: []
I1014 19:27:38.599799   842 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 664.544, Next index: 545, Last known committed idx: 544, Time since last communication: 0.000s
I1014 19:27:38.600093   841 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 LEADER]: Stepping down as leader of term 664
I1014 19:27:38.600148   841 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:38.600406   841 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 544, Committed index: 544, Last appended: 664.544, Last appended by leader: 544, Current term: 664, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.600662   841 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:38.600698   841 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:38.600809   841 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 664 FOLLOWER]: Advancing to term 665
I1014 19:27:38.670150 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:38.670243 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:38.670305 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 664 FOLLOWER]: Advancing to term 665
I1014 19:27:38.672452 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.672966 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 665 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.673141 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 665 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.673740 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 665 candidate_status { last_received { term: 664 index: 544 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:38.673933 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 665: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:38.673974 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 665 candidate_status { last_received { term: 664 index: 544 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:38.674108 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:38.674352 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 665 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 665: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:38.675643 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 665.
I1014 19:27:38.675956 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 665 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.676014 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 665 election: Election decided. Result: candidate won.
I1014 19:27:38.676335 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.103s (election complete)
I1014 19:27:38.676458 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 FOLLOWER]: Leader election won for term 665
I1014 19:27:38.677362 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:38.677621 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 544, Committed index: 544, Last appended: 664.544, Last appended by leader: 544, Current term: 665, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.678752   908 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.679039   908 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.679559   909 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:38.681126   909 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:38.682116   909 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:38.682790   909 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:38.684628 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 664 index: 544. Preceding OpId from leader: term: 665 index: 546. (index mismatch)
I1014 19:27:38.684656 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 664 index: 544. Preceding OpId from leader: term: 665 index: 546. (index mismatch)
I1014 19:27:38.685138   908 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 545, Last known committed idx: 544, Time since last communication: 0.000s
I1014 19:27:38.685349 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 545, Last known committed idx: 544, Time since last communication: 0.000s
I1014 19:27:38.686759   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.687022   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.687319   841 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.687572   841 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.688930   841 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.688943   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.689098   841 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.689121   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.690212 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.690614 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.690204   910 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 665 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.690851   910 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.789561   665 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:38.789650   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:38.789713   665 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 665 FOLLOWER]: Advancing to term 666
I1014 19:27:38.791301   665 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.791867   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 666 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.792026   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 666 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:38.792699 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 666 candidate_status { last_received { term: 665 index: 546 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:38.792886 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 666 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.793154 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 666 candidate_status { last_received { term: 665 index: 546 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.793274 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 666 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 666 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.793331 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 666 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.793654 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 666 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 666 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:38.793721 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 666 election: Election decided. Result: candidate lost.
I1014 19:27:38.794781 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 665. Current term is 666. Ops: []
I1014 19:27:38.795233   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:38.795295   665 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Leader election lost for term 666. Reason: could not achieve majority
I1014 19:27:38.795269   910 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 665.546, Next index: 547, Last known committed idx: 546, Time since last communication: 0.000s
I1014 19:27:38.795531 32224 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 LEADER]: Stepping down as leader of term 665
I1014 19:27:38.795578 32224 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:38.795756 32224 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 546, Committed index: 546, Last appended: 665.546, Last appended by leader: 546, Current term: 665, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.795917 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:38.795950 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:38.796056 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 665 FOLLOWER]: Advancing to term 666
I1014 19:27:38.878733   841 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:38.878814   841 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:38.878861   841 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 665 FOLLOWER]: Advancing to term 666
I1014 19:27:38.881180   841 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.881729   841 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 666 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:38.881881   841 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 666 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.882442 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 666 candidate_status { last_received { term: 665 index: 546 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:38.882436 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 666 candidate_status { last_received { term: 665 index: 546 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:38.882612 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (vote granted)
I1014 19:27:38.882637 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 666: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:38.882994 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 666 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 666: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:38.884600 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 666 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 666.
I1014 19:27:38.884938 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 666 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:38.884992 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 666 election: Election decided. Result: candidate won.
I1014 19:27:38.885174   841 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:38.885227   841 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 FOLLOWER]: Leader election won for term 666
I1014 19:27:38.885313   841 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:38.885538   841 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 546, Committed index: 546, Last appended: 665.546, Last appended by leader: 546, Current term: 666, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:38.886445   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.886780   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.886943   840 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:38.888386   840 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:38.889389   840 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:38.890242   840 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:38.892423 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 666 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 665 index: 546. Preceding OpId from leader: term: 666 index: 548. (index mismatch)
I1014 19:27:38.892586 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 665 index: 546. Preceding OpId from leader: term: 666 index: 548. (index mismatch)
I1014 19:27:38.892990   841 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 547, Last known committed idx: 546, Time since last communication: 0.000s
I1014 19:27:38.893143   842 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 547, Last known committed idx: 546, Time since last communication: 0.000s
I1014 19:27:38.894608 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.894793 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.894948   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.895139   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.896546   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.896610   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.896731   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.896759   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:38.897462   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.897650   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.898062 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 666 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:38.898231 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:38.997511   665 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:38.997596   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:38.997645   665 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 666 FOLLOWER]: Advancing to term 667
I1014 19:27:38.999783   665 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.000234 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 666. Current term is 667. Ops: []
I1014 19:27:39.000258   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 667 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.000435   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 667 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.000877   839 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 666.548, Next index: 549, Last known committed idx: 548, Time since last communication: 0.000s
I1014 19:27:39.001118   839 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 LEADER]: Stepping down as leader of term 666
I1014 19:27:39.001101 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 667 candidate_status { last_received { term: 666 index: 548 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.001217   839 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:39.001482   839 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 548, Committed index: 548, Last appended: 666.548, Last appended by leader: 548, Current term: 666, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.001678   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:39.001713   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:39.001821   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 666 FOLLOWER]: Advancing to term 667
I1014 19:27:39.003914 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:39.004289 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 667 candidate_status { last_received { term: 666 index: 548 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.004496 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 666 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 667 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.004868 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 667 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 666 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 667 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.005848 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 667.
I1014 19:27:39.006407 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 667 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.006462 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 667 election: Election decided. Result: candidate won.
I1014 19:27:39.006669   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (election complete)
I1014 19:27:39.006726   665 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 FOLLOWER]: Leader election won for term 667
I1014 19:27:39.007017   665 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:39.007223   665 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 548, Committed index: 548, Last appended: 666.548, Last appended by leader: 548, Current term: 667, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.008330  1032 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.008555  1032 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.013356  1033 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:39.014688  1033 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:39.015646  1033 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:39.016688  1033 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:39.019207 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 666 FOLLOWER]: Advancing to term 667
I1014 19:27:39.019232 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 666 index: 548. Preceding OpId from leader: term: 667 index: 550. (index mismatch)
I1014 19:27:39.019768   665 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 549, Last known committed idx: 548, Time since last communication: 0.000s
I1014 19:27:39.021415 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 667 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 666 index: 548. Preceding OpId from leader: term: 667 index: 550. (index mismatch)
I1014 19:27:39.021303   841 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.021571   841 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.021931   665 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 549, Last known committed idx: 548, Time since last communication: 0.000s
I1014 19:27:39.024667 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.024853 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.025745   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.025929   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.027037 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.027215 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.027837   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.027998   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.029186   841 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 667 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.029376   841 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.126704 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 667 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:39.126780 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:39.126829 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 667 FOLLOWER]: Advancing to term 668
I1014 19:27:39.128659 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.129060 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 668 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.129206 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 668 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.129619 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 668 candidate_status { last_received { term: 667 index: 550 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.129806 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 668 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.129789 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 668 candidate_status { last_received { term: 667 index: 550 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.129994 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 668 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.130342 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 668 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 668 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.130481 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 668 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 668 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.130556 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 668 election: Election decided. Result: candidate lost.
I1014 19:27:39.130791 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:39.130848 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Leader election lost for term 668. Reason: could not achieve majority
I1014 19:27:39.132673 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 667. Current term is 668. Ops: []
I1014 19:27:39.133101  1040 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 667.550, Next index: 551, Last known committed idx: 550, Time since last communication: 0.000s
I1014 19:27:39.133297  1040 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 LEADER]: Stepping down as leader of term 667
I1014 19:27:39.133335  1040 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:39.133534  1040 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 550, Committed index: 550, Last appended: 667.550, Last appended by leader: 550, Current term: 667, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.133708  1040 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:39.133749  1040 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:39.133852  1040 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 667 FOLLOWER]: Advancing to term 668
I1014 19:27:39.221923   841 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:39.221992   841 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.113s (starting election)
I1014 19:27:39.222039   841 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 667 FOLLOWER]: Advancing to term 668
I1014 19:27:39.223577   841 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.223954   841 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 668 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.224097   841 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 668 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.224632 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 668 candidate_status { last_received { term: 667 index: 550 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.224661 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 668 candidate_status { last_received { term: 667 index: 550 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.224756 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.105s (vote granted)
I1014 19:27:39.224829 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 668: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:39.225114 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 668 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 668: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:39.226084 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 668 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 668.
I1014 19:27:39.226339 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 668 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.226392 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 668 election: Election decided. Result: candidate won.
I1014 19:27:39.226578   841 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:39.226634   841 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 FOLLOWER]: Leader election won for term 668
I1014 19:27:39.226723   841 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:39.226884   841 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 550, Committed index: 550, Last appended: 667.550, Last appended by leader: 550, Current term: 668, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.227775   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.228029   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.228191   840 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:39.229687   840 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:39.231083   840 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:39.232203   840 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:39.233839 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 667 index: 550. Preceding OpId from leader: term: 668 index: 552. (index mismatch)
I1014 19:27:39.233925 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 668 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 667 index: 550. Preceding OpId from leader: term: 668 index: 552. (index mismatch)
I1014 19:27:39.234238   839 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 551, Last known committed idx: 550, Time since last communication: 0.000s
I1014 19:27:39.234463   841 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 551, Last known committed idx: 550, Time since last communication: 0.000s
I1014 19:27:39.235747 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.235926 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.235740   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.235967   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.237156   469 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.237396   469 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.237386   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.237679   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.237952   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.238093   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.238482 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 668 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.238653 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.338213   665 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 668 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:39.338302   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:39.338353   665 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 668 FOLLOWER]: Advancing to term 669
I1014 19:27:39.340538   665 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.341075   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 669 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.341261   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 669 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.342069 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 669 candidate_status { last_received { term: 668 index: 552 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.342070 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 669 candidate_status { last_received { term: 668 index: 552 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.342257 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 669 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.342259 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 669 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.342622 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 669 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 669 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.342710 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 669 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 669 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.342774 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 669 election: Election decided. Result: candidate lost.
I1014 19:27:39.342960   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:39.343111   665 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Leader election lost for term 669. Reason: could not achieve majority
I1014 19:27:39.355792 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 668. Current term is 669. Ops: []
I1014 19:27:39.356493   839 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 668.552, Next index: 553, Last known committed idx: 552, Time since last communication: 0.000s
I1014 19:27:39.356792   839 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 LEADER]: Stepping down as leader of term 668
I1014 19:27:39.356832   839 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:39.357086   839 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 552, Committed index: 552, Last appended: 668.552, Last appended by leader: 552, Current term: 668, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.357262   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:39.357304   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:39.357427   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 668 FOLLOWER]: Advancing to term 669
I1014 19:27:39.419435 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:39.419551 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:39.419610 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 668 FOLLOWER]: Advancing to term 669
I1014 19:27:39.421064 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.421465 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 669 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.421634 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 669 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.422154 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 669 candidate_status { last_received { term: 668 index: 552 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.422153 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 669 candidate_status { last_received { term: 668 index: 552 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.422325 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 669: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:39.422343 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:39.422770 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 669 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 669: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:39.424448 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 669 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 669.
I1014 19:27:39.424767 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 669 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.424819 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 669 election: Election decided. Result: candidate won.
I1014 19:27:39.425000 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.140s (election complete)
I1014 19:27:39.425058 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 FOLLOWER]: Leader election won for term 669
I1014 19:27:39.425330 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:39.425556 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 552, Committed index: 552, Last appended: 668.552, Last appended by leader: 552, Current term: 669, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.426571  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.426756  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.427147  1172 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:39.428519  1172 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:39.429402  1172 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:39.430048  1172 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:39.432241 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 668 index: 552. Preceding OpId from leader: term: 669 index: 554. (index mismatch)
I1014 19:27:39.432232 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 669 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 668 index: 552. Preceding OpId from leader: term: 669 index: 554. (index mismatch)
I1014 19:27:39.432672  1171 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 553, Last known committed idx: 552, Time since last communication: 0.000s
I1014 19:27:39.434067   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.434226   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.435065  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.436036  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.436144   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.436316   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.443089  1171 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 553, Last known committed idx: 552, Time since last communication: 0.000s
I1014 19:27:39.447928   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.448134   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.448308   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.448477   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.546631   839 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 669 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:39.546708   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:39.546761   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 669 FOLLOWER]: Advancing to term 670
I1014 19:27:39.548810   839 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.549296   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 670 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.549448   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 670 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.549962 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 670 candidate_status { last_received { term: 669 index: 554 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.549965 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 670 candidate_status { last_received { term: 669 index: 554 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.550118 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 670 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.550135 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 670 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.550467 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 670 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 670 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.550626 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 670 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 670 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.550678 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 670 election: Election decided. Result: candidate lost.
I1014 19:27:39.550833   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:39.550889   839 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Leader election lost for term 670. Reason: could not achieve majority
I1014 19:27:39.564404  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 669 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.564636  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.564910 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 669. Current term is 670. Ops: []
I1014 19:27:39.565419 32224 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 669.554, Next index: 555, Last known committed idx: 554, Time since last communication: 0.000s
I1014 19:27:39.565631 32224 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 LEADER]: Stepping down as leader of term 669
I1014 19:27:39.565670 32224 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:39.565831 32224 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 554, Committed index: 554, Last appended: 669.554, Last appended by leader: 554, Current term: 669, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.566056 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:39.566097 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:39.566208 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 669 FOLLOWER]: Advancing to term 670
I1014 19:27:39.613534   665 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:39.613617   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:39.613689   665 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 669 FOLLOWER]: Advancing to term 670
I1014 19:27:39.615075   665 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.615429   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 670 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.615586   665 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 670 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.616026 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 670 candidate_status { last_received { term: 669 index: 554 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.616112 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 670 candidate_status { last_received { term: 669 index: 554 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.616188 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (vote granted)
I1014 19:27:39.616284 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 670: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:39.616705 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 670 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 670: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:39.617723 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 670 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 670.
I1014 19:27:39.618026 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 670 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.618067 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 670 election: Election decided. Result: candidate won.
I1014 19:27:39.618211   665 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (election complete)
I1014 19:27:39.618274   665 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 FOLLOWER]: Leader election won for term 670
I1014 19:27:39.618357   665 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:39.618530   665 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 554, Committed index: 554, Last appended: 669.554, Last appended by leader: 554, Current term: 670, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.619220  1040 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.619346  1040 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.619774  1277 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:39.621266  1277 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:39.622231  1277 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:39.623098  1277 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:39.625139 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 670 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 669 index: 554. Preceding OpId from leader: term: 670 index: 556. (index mismatch)
I1014 19:27:39.625140 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 669 index: 554. Preceding OpId from leader: term: 670 index: 556. (index mismatch)
I1014 19:27:39.625663  1040 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 555, Last known committed idx: 554, Time since last communication: 0.000s
I1014 19:27:39.625926  1040 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 555, Last known committed idx: 554, Time since last communication: 0.000s
I1014 19:27:39.627111 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.627295 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.627378   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.627602   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.628506  1040 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.628691  1040 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.628787   665 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.628973   665 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.629494 25848 mvcc.cc:203] Tried to move safe_time back from 6306080602620592128 to 6306080602599387136. Current Snapshot: MvccSnapshot[committed={T|T < 6306080602620592128}]
I1014 19:27:39.629715 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.629884 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.629896   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 670 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.630127   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.730226 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 670 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:39.730301 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:39.730353 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 670 FOLLOWER]: Advancing to term 671
I1014 19:27:39.732334 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.732764 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 671 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.732905 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 671 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.733373 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 671 candidate_status { last_received { term: 670 index: 556 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.733583 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 671 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.733482 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 671 candidate_status { last_received { term: 670 index: 556 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.733675 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 671 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.733973 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 671 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 671 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.734519 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 671 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 671 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.734585 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 671 election: Election decided. Result: candidate lost.
I1014 19:27:39.734866 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:39.734920 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Leader election lost for term 671. Reason: could not achieve majority
I1014 19:27:39.747664 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 670. Current term is 671. Ops: []
I1014 19:27:39.748778  1279 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 670.556, Next index: 557, Last known committed idx: 556, Time since last communication: 0.000s
I1014 19:27:39.749029  1279 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 LEADER]: Stepping down as leader of term 670
I1014 19:27:39.749070  1279 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:39.749317  1279 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 556, Committed index: 556, Last appended: 670.556, Last appended by leader: 556, Current term: 670, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.749580  1279 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:39.749624  1279 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:39.749765  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 670 FOLLOWER]: Advancing to term 671
I1014 19:27:39.823583   839 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:39.823668   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.140s (starting election)
I1014 19:27:39.823730   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 670 FOLLOWER]: Advancing to term 671
I1014 19:27:39.825237   839 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.825678   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 671 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.825837   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 671 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.826422 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 671 candidate_status { last_received { term: 670 index: 556 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.826422 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 671 candidate_status { last_received { term: 670 index: 556 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:39.826614 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (vote granted)
I1014 19:27:39.826642 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 671: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:39.827682 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 671 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 671: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:39.827970 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 671 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 671.
I1014 19:27:39.828222 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 671 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:39.828269 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 671 election: Election decided. Result: candidate won.
I1014 19:27:39.828435   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:39.828491   839 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 FOLLOWER]: Leader election won for term 671
I1014 19:27:39.828619   839 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:39.828780   839 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 556, Committed index: 556, Last appended: 670.556, Last appended by leader: 556, Current term: 671, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.829915   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.830189   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.830564  1350 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:39.831939  1350 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:39.832867  1350 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:39.833514  1350 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:39.835266 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 671 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 670 index: 556. Preceding OpId from leader: term: 671 index: 558. (index mismatch)
I1014 19:27:39.835266 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 670 index: 556. Preceding OpId from leader: term: 671 index: 558. (index mismatch)
I1014 19:27:39.835741   842 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 557, Last known committed idx: 556, Time since last communication: 0.000s
I1014 19:27:39.835966   839 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 557, Last known committed idx: 556, Time since last communication: 0.000s
I1014 19:27:39.837304 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.837301  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.837528 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.837563  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.838984   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.839162   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.839295 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.839484 25850 mvcc.cc:203] Tried to move safe_time back from 6306080603480944640 to 6306080603461980160. Current Snapshot: MvccSnapshot[committed={T|T < 6306080603480944640}]
I1014 19:27:39.839538 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.839423  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.839617  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:39.839045   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 671 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:39.839941   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:39.939337  1279 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 671 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:39.939419  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (starting election)
I1014 19:27:39.939471  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 671 FOLLOWER]: Advancing to term 672
I1014 19:27:39.941663  1279 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.942159  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 672 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:39.942319  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 672 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:39.943011 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 672 candidate_status { last_received { term: 671 index: 558 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:39.943197 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 672 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.943570 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 672 candidate_status { last_received { term: 671 index: 558 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:39.943667 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 672 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 672 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.943758 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 672 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.944151 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 672 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 672 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:39.944202 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 672 election: Election decided. Result: candidate lost.
I1014 19:27:39.944463  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.133s (election complete)
I1014 19:27:39.944538  1279 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Leader election lost for term 672. Reason: could not achieve majority
I1014 19:27:39.946861 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 671. Current term is 672. Ops: []
I1014 19:27:39.947253   842 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 671.558, Next index: 559, Last known committed idx: 558, Time since last communication: 0.000s
I1014 19:27:39.947517   839 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 LEADER]: Stepping down as leader of term 671
I1014 19:27:39.947561   839 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:39.947721   839 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 558, Committed index: 558, Last appended: 671.558, Last appended by leader: 558, Current term: 671, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:39.947845   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:39.947871   839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:39.947960   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 671 FOLLOWER]: Advancing to term 672
I1014 19:27:40.016144 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:40.016216 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.105s (starting election)
I1014 19:27:40.016270 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 671 FOLLOWER]: Advancing to term 672
I1014 19:27:40.017644 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.018062 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 672 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:40.018203 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 672 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.018679 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 672 candidate_status { last_received { term: 671 index: 558 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:40.018764 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 672 candidate_status { last_received { term: 671 index: 558 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.018877 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 672: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.018930 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (vote granted)
I1014 19:27:40.019217 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 672 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 672: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.020339 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 672.
I1014 19:27:40.020612 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 672 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.020660 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 672 election: Election decided. Result: candidate won.
I1014 19:27:40.020944 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:40.020999 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 FOLLOWER]: Leader election won for term 672
I1014 19:27:40.021073 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:40.021245 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 558, Committed index: 558, Last appended: 671.558, Last appended by leader: 558, Current term: 672, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.021981  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.022104  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.022261  1172 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:40.023640  1172 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:40.024581  1172 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:40.025425  1172 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:40.027403 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 671 index: 558. Preceding OpId from leader: term: 672 index: 560. (index mismatch)
I1014 19:27:40.027532 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 671 index: 558. Preceding OpId from leader: term: 672 index: 560. (index mismatch)
I1014 19:27:40.027925 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 559, Last known committed idx: 558, Time since last communication: 0.000s
I1014 19:27:40.028116  1171 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 559, Last known committed idx: 558, Time since last communication: 0.000s
I1014 19:27:40.029435  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.029580  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.030205   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.030380   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.030867  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.031059  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.031663  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.031884  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.032579  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.032784  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.032999   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 672 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.033208   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.133316  1279 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:40.133399  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:40.133445  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 672 FOLLOWER]: Advancing to term 673
I1014 19:27:40.135779  1279 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.136482  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 673 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.137063  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 673 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.137275 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 673 candidate_status { last_received { term: 672 index: 560 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.137517 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 673 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.138084 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 673 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 673 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.138252 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 673 candidate_status { last_received { term: 672 index: 560 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.138480 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 673 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.138823 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 673 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 673 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.138890 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 673 election: Election decided. Result: candidate lost.
I1014 19:27:40.139086  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:40.139144  1279 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Leader election lost for term 673. Reason: could not achieve majority
I1014 19:27:40.157028 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 672. Current term is 673. Ops: []
I1014 19:27:40.157661  1408 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 672.560, Next index: 561, Last known committed idx: 560, Time since last communication: 0.000s
I1014 19:27:40.158012  1171 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 LEADER]: Stepping down as leader of term 672
I1014 19:27:40.158066  1171 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:40.158334  1171 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 560, Committed index: 560, Last appended: 672.560, Last appended by leader: 560, Current term: 672, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.158550  1171 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:40.158591  1171 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:40.158715  1171 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 672 FOLLOWER]: Advancing to term 673
I1014 19:27:40.232633   839 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:40.232710   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:40.232759   839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 672 FOLLOWER]: Advancing to term 673
I1014 19:27:40.234241   839 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.234658   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 673 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:40.234802   839 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 673 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.235286 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 673 candidate_status { last_received { term: 672 index: 560 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:40.235292 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 673 candidate_status { last_received { term: 672 index: 560 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.235477 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 673: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.235476 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:40.235893 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 673 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 673: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.236932 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 673.
I1014 19:27:40.237200 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 673 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.237246 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 673 election: Election decided. Result: candidate won.
I1014 19:27:40.237390   839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (election complete)
I1014 19:27:40.237447   839 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 FOLLOWER]: Leader election won for term 673
I1014 19:27:40.237557   839 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:40.237720   839 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 560, Committed index: 560, Last appended: 672.560, Last appended by leader: 560, Current term: 673, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.238598   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.238796   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.238946  1350 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:40.240236  1350 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:40.241243  1350 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:40.241909  1350 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:40.244165 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 672 index: 560. Preceding OpId from leader: term: 673 index: 562. (index mismatch)
I1014 19:27:40.244577 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 672 index: 560. Preceding OpId from leader: term: 673 index: 562. (index mismatch)
I1014 19:27:40.244699   839 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 561, Last known committed idx: 560, Time since last communication: 0.001s
I1014 19:27:40.245337   842 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 561, Last known committed idx: 560, Time since last communication: 0.000s
I1014 19:27:40.246857  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.247092  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.247092  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.247267  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.247637   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.247766   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.248594   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.248771   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.248872  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.249027  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.249653  1171 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 673 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.249830  1171 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.348927  1279 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:40.349000  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:40.349050  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 673 FOLLOWER]: Advancing to term 674
I1014 19:27:40.351109  1279 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.351577  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 674 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.351721  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 674 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.352120 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 674 candidate_status { last_received { term: 673 index: 562 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.352286 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 674 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.352564 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 674 candidate_status { last_received { term: 673 index: 562 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.352735 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 674 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.353137 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 674 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 674 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.353258 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 674 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 674 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.353328 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 674 election: Election decided. Result: candidate lost.
I1014 19:27:40.354044  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:40.354106  1279 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Leader election lost for term 674. Reason: could not achieve majority
I1014 19:27:40.364410 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 673. Current term is 674. Ops: []
I1014 19:27:40.364989  1492 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 673.562, Next index: 563, Last known committed idx: 562, Time since last communication: 0.000s
I1014 19:27:40.365244  1492 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 LEADER]: Stepping down as leader of term 673
I1014 19:27:40.365278  1492 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:40.365453  1492 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 562, Committed index: 562, Last appended: 673.562, Last appended by leader: 562, Current term: 673, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.365661  1492 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:40.365705  1492 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:40.365839  1492 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 673 FOLLOWER]: Advancing to term 674
I1014 19:27:40.437916  1171 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:40.438009  1171 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:40.438062  1171 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 673 FOLLOWER]: Advancing to term 674
I1014 19:27:40.440171  1171 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.440691  1171 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 674 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:40.440856  1171 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 674 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.441381 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 674 candidate_status { last_received { term: 673 index: 562 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:40.441565 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 674: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.441459 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 674 candidate_status { last_received { term: 673 index: 562 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.441642 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.132s (vote granted)
I1014 19:27:40.441910 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 674 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 674: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.443768 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 674.
I1014 19:27:40.444077 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 674 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.444134 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 674 election: Election decided. Result: candidate won.
I1014 19:27:40.444574  1171 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:40.444634  1171 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 FOLLOWER]: Leader election won for term 674
I1014 19:27:40.444716  1171 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:40.444898  1171 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 562, Committed index: 562, Last appended: 673.562, Last appended by leader: 562, Current term: 674, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.446195  1408 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.446475  1408 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.446666  1172 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:40.448426  1172 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:40.449704  1172 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:40.450495  1172 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:40.452456 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 673 index: 562. Preceding OpId from leader: term: 674 index: 564. (index mismatch)
I1014 19:27:40.452639 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 673 index: 562. Preceding OpId from leader: term: 674 index: 564. (index mismatch)
I1014 19:27:40.452878  1408 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 563, Last known committed idx: 562, Time since last communication: 0.000s
I1014 19:27:40.453413  1408 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 563, Last known committed idx: 562, Time since last communication: 0.000s
I1014 19:27:40.454461  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.454679  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.455152   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.455353   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.455696 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.455946 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.456091 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.456245 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.456274  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.456441  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.458109   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 674 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.458281   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.556895  1279 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:40.556964  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (starting election)
I1014 19:27:40.557013  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 674 FOLLOWER]: Advancing to term 675
I1014 19:27:40.558526  1279 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.558938  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 675 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.559096  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 675 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.559499 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 675 candidate_status { last_received { term: 674 index: 564 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.559592 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 675 candidate_status { last_received { term: 674 index: 564 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.559707 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 675 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.559762 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 675 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.560036 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 675 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 675 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.560118 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 675 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 675 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.560194 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 675 election: Election decided. Result: candidate lost.
I1014 19:27:40.560330  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (election complete)
I1014 19:27:40.560380  1279 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Leader election lost for term 675. Reason: could not achieve majority
I1014 19:27:40.572929 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 674. Current term is 675. Ops: []
I1014 19:27:40.573403  1171 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 674.564, Next index: 565, Last known committed idx: 564, Time since last communication: 0.000s
I1014 19:27:40.573658 32224 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 LEADER]: Stepping down as leader of term 674
I1014 19:27:40.573709 32224 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:40.573882 32224 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 564, Committed index: 564, Last appended: 674.564, Last appended by leader: 564, Current term: 674, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.574059 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:40.574095 32224 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:40.574204 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 674 FOLLOWER]: Advancing to term 675
I1014 19:27:40.651964   842 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:40.652055   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:40.652112   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 674 FOLLOWER]: Advancing to term 675
I1014 19:27:40.653739   842 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.654145   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 675 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:40.654285   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 675 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.655002 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 675 candidate_status { last_received { term: 674 index: 564 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:40.655002 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 675 candidate_status { last_received { term: 674 index: 564 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.655174 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:40.655205 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 675: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.655612 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 675 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 675: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:40.656692 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 675 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 675.
I1014 19:27:40.657052 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 675 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.657109 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 675 election: Election decided. Result: candidate won.
I1014 19:27:40.657296   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:40.657397   842 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 FOLLOWER]: Leader election won for term 675
I1014 19:27:40.657500   842 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:40.657766   842 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 564, Committed index: 564, Last appended: 674.564, Last appended by leader: 564, Current term: 675, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.658790  1492 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.659037  1492 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.659240  1350 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:40.660714  1350 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:40.661089 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:40.661787  1350 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:40.662406  1350 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:40.664305 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 675 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 674 index: 564. Preceding OpId from leader: term: 675 index: 566. (index mismatch)
I1014 19:27:40.664305 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 674 index: 564. Preceding OpId from leader: term: 675 index: 566. (index mismatch)
I1014 19:27:40.664752   842 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 565, Last known committed idx: 564, Time since last communication: 0.000s
I1014 19:27:40.664927  1492 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 565, Last known committed idx: 564, Time since last communication: 0.000s
I1014 19:27:40.666563 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.666757 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.666769  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.666944  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.667796   839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.667976   839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.667899   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.668308   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.668931  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.669108  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.669260 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 675 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.669615 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.854614 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 675 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:40.854682 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:40.854732 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 675 FOLLOWER]: Advancing to term 676
I1014 19:27:40.856083 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.856510 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 676 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:40.856644 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 676 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.857097 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 676 candidate_status { last_received { term: 675 index: 566 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:40.857264 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 676 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.857311 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 676 candidate_status { last_received { term: 675 index: 566 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.857508 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 676 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.857576 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 676 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 676 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.857842 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 676 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 676 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:40.857905 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 676 election: Election decided. Result: candidate lost.
I1014 19:27:40.858083 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.124s (election complete)
I1014 19:27:40.858148 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election lost for term 676. Reason: could not achieve majority
I1014 19:27:40.876883 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 675. Current term is 676. Ops: []
I1014 19:27:40.877362   839 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 675.566, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I1014 19:27:40.877645   842 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 LEADER]: Stepping down as leader of term 675
I1014 19:27:40.877684   842 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:40.877847   842 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 566, Committed index: 566, Last appended: 675.566, Last appended by leader: 566, Current term: 675, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.877988   842 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:40.878028   842 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:40.878170   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 675 FOLLOWER]: Advancing to term 676
I1014 19:27:40.944489  1279 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:40.944579  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.126s (starting election)
I1014 19:27:40.944634  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 675 FOLLOWER]: Advancing to term 676
I1014 19:27:40.946056  1279 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.946523  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 676 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:40.946676  1279 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 676 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.947240 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 676 candidate_status { last_received { term: 675 index: 566 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:40.947281 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 676 candidate_status { last_received { term: 675 index: 566 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:40.947424 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (vote granted)
I1014 19:27:40.947444 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 676: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:40.947803 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 676 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 676: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:40.948933 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 676 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 676.
I1014 19:27:40.949184 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 676 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:40.949231 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 676 election: Election decided. Result: candidate won.
I1014 19:27:40.949390  1279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.144s (election complete)
I1014 19:27:40.949450  1279 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 FOLLOWER]: Leader election won for term 676
I1014 19:27:40.949796  1279 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:40.949992  1279 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 566, Committed index: 566, Last appended: 675.566, Last appended by leader: 566, Current term: 676, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:40.951035  1780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.951376  1780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.951881  1781 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:40.953258  1781 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:40.954182  1781 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:40.954792  1781 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:40.956662 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 676 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 675 index: 566. Preceding OpId from leader: term: 676 index: 568. (index mismatch)
I1014 19:27:40.956764 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 675 index: 566. Preceding OpId from leader: term: 676 index: 568. (index mismatch)
I1014 19:27:40.957115  1279 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I1014 19:27:40.957311  1780 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 567, Last known committed idx: 566, Time since last communication: 0.000s
I1014 19:27:40.958734   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.958938   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.959735 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.960130 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.960619   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.960849   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:40.961493  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.961872  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.962036  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.962247  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:40.962256 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 676 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:40.962427 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.060878   842 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 676 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:41.060956   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (starting election)
I1014 19:27:41.061066   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 676 FOLLOWER]: Advancing to term 677
I1014 19:27:41.063249   842 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 677 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.063781   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 677 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.063941   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 677 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.064589 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 677 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.064756 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.065117 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 677 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.065313 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 677 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.065486 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.065690 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 677 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 676. Current term is 677. Ops: []
I1014 19:27:41.065783 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 677 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.065837 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 677 election: Election decided. Result: candidate lost.
I1014 19:27:41.066109   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.104s (election complete)
I1014 19:27:41.066166   842 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 677 FOLLOWER]: Leader election lost for term 677. Reason: could not achieve majority
I1014 19:27:41.066159  1279 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 676.568, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I1014 19:27:41.066375  1279 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 LEADER]: Stepping down as leader of term 676
I1014 19:27:41.066406  1279 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:41.066581  1279 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 568, Committed index: 568, Last appended: 676.568, Last appended by leader: 568, Current term: 676, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.066745  1279 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:41.066776  1279 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:41.066876  1279 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 676 FOLLOWER]: Advancing to term 677
I1014 19:27:41.149662  1782 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 677 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:41.149811  1782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:41.149870  1782 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 677 FOLLOWER]: Advancing to term 678
I1014 19:27:41.151294  1782 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.151726  1782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 678 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.151850  1782 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 678 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.152377 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 678 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.152380 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 678 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.152513 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 677 FOLLOWER]: Advancing to term 678
I1014 19:27:41.152552 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 678 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.152578 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.191s (vote granted)
I1014 19:27:41.152937 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 678 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 678 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.154122 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 678.
I1014 19:27:41.154400 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 678 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.154459 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 678 election: Election decided. Result: candidate won.
I1014 19:27:41.154621  1782 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:41.154685  1782 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 FOLLOWER]: Leader election won for term 678
I1014 19:27:41.154772  1782 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:41.154939  1782 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 568, Committed index: 568, Last appended: 676.568, Last appended by leader: 568, Current term: 678, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.155891  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.156127  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.156215 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:41.156277 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:41.156306  1781 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:41.156325 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 676 FOLLOWER]: Advancing to term 677
I1014 19:27:41.157670  1781 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:41.157795 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 677 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.158243 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 677 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.158396 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 677 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.159023  1781 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:41.159335 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 677 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.159529 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 677. Current term is 678.
I1014 19:27:41.159540 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 677 candidate_status { last_received { term: 676 index: 568 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.159773 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.159862  1781 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
W1014 19:27:41.160176 28780 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 677 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 677. Current term is 678.
I1014 19:27:41.160207 28780 leader_election.cc:374] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 677 election: Cancelling election due to peer responding with higher term
W1014 19:27:41.160367 28786 leader_election.cc:371] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 677 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 677 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.160385 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:41.160465 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 677 FOLLOWER]: Advancing to term 678
I1014 19:27:41.161840 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 676 index: 568. Preceding OpId from leader: term: 678 index: 570. (index mismatch)
I1014 19:27:41.161936 32224 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Leader election lost for term 677. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for earlier term 677. Current term is 678.
I1014 19:27:41.162091 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 676 index: 568. Preceding OpId from leader: term: 678 index: 570. (index mismatch)
I1014 19:27:41.162503  1279 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I1014 19:27:41.162659  1782 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I1014 19:27:41.164585   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.164878   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.165398  1780 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.165609  1780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.164376 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.165895 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.166685 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.166930 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.167452  1279 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.167631  1279 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.167500   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 678 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.167726   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.367338   842 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:41.367421   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:41.367475   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 678 FOLLOWER]: Advancing to term 679
I1014 19:27:41.369810   842 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.370309   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 679 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.370466   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 679 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.370913 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 679 candidate_status { last_received { term: 678 index: 570 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.371001 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 679 candidate_status { last_received { term: 678 index: 570 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.371104 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 679 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.371173 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 679 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.371547 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 679 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 679 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.371824 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 679 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 679 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.371876 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 679 election: Election decided. Result: candidate lost.
I1014 19:27:41.372370   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (election complete)
I1014 19:27:41.372454   842 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Leader election lost for term 679. Reason: could not achieve majority
I1014 19:27:41.380103 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 678. Current term is 679. Ops: []
I1014 19:27:41.380728  1839 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 678.570, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I1014 19:27:41.381067  1839 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Stepping down as leader of term 678
I1014 19:27:41.381158  1839 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:41.381332  1839 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 570, Committed index: 570, Last appended: 678.570, Last appended by leader: 570, Current term: 678, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.381558  1839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:41.381595  1839 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:41.381701  1839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 678 FOLLOWER]: Advancing to term 679
I1014 19:27:41.452235 32224 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:41.452311 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.144s (starting election)
I1014 19:27:41.452365 32224 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 678 FOLLOWER]: Advancing to term 679
I1014 19:27:41.453924 32224 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.454293 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 679 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.454427 32224 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 679 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.454898 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 679 candidate_status { last_received { term: 678 index: 570 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.454977 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 679 candidate_status { last_received { term: 678 index: 570 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.455057 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.134s (vote granted)
I1014 19:27:41.455164 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 679: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:41.455562 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 679 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 679: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:41.456559 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 679.
I1014 19:27:41.456881 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 679 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.456928 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 679 election: Election decided. Result: candidate won.
I1014 19:27:41.457070 32224 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:41.457125 32224 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 FOLLOWER]: Leader election won for term 679
I1014 19:27:41.457382 32224 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:41.457602 32224 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 570, Committed index: 570, Last appended: 678.570, Last appended by leader: 570, Current term: 679, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.458387  2010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.458616  2010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.458969  2011 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:41.460414  2011 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:41.461510  2011 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:41.462956  2011 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:41.464627 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 678 index: 570. Preceding OpId from leader: term: 679 index: 572. (index mismatch)
I1014 19:27:41.464639 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 678 index: 570. Preceding OpId from leader: term: 679 index: 572. (index mismatch)
I1014 19:27:41.465032 32224 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I1014 19:27:41.465330  2010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I1014 19:27:41.466445   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.466683   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.466697  1839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.467007  1839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.468464   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.468693   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.468695 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.468897 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.468786  2012 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.469089  2012 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.469880  1839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 679 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.470088  1839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.569262   842 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:41.569355   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (starting election)
I1014 19:27:41.569430   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 679 FOLLOWER]: Advancing to term 680
I1014 19:27:41.572007   842 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 680 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.572554   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 680 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.572803   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 680 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.573303 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 680 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.573395 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 680 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.573526 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 680 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.573604 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 680 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.573923 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 680 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 680 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.574123 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 680 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 680 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.574198 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 680 election: Election decided. Result: candidate lost.
I1014 19:27:41.574429   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:41.574596   842 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 680 FOLLOWER]: Leader election lost for term 680. Reason: could not achieve majority
I1014 19:27:41.578157 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 680 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 679. Current term is 680. Ops: []
I1014 19:27:41.578610  2012 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 679.572, Next index: 573, Last known committed idx: 572, Time since last communication: 0.000s
I1014 19:27:41.578801  2012 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 LEADER]: Stepping down as leader of term 679
I1014 19:27:41.578847  2012 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:41.579023  2012 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 572, Committed index: 572, Last appended: 679.572, Last appended by leader: 572, Current term: 679, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.579197  2012 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:41.579238  2012 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:41.579341  2012 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 679 FOLLOWER]: Advancing to term 680
I1014 19:27:41.665143  1839 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:41.665216  1839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (starting election)
I1014 19:27:41.665266  1839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 679 FOLLOWER]: Advancing to term 680
I1014 19:27:41.667158  2010 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 680 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:41.667227  2010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (starting election)
I1014 19:27:41.667279  2010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 680 FOLLOWER]: Advancing to term 681
I1014 19:27:41.669098  2010 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.669562  2010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 681 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.669700  2010 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 681 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.670050 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 681 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.670207 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 681 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.670342 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 680 FOLLOWER]: Advancing to term 681
I1014 19:27:41.670424 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (vote granted)
I1014 19:27:41.671027  1839 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 680 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.671479  1839 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 680 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.671576 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 680 FOLLOWER]: Advancing to term 681
I1014 19:27:41.671700 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.124s (vote granted)
I1014 19:27:41.671838 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 681.
I1014 19:27:41.671700  1839 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 680 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.672109 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 681 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.672147 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 681 election: Election decided. Result: candidate won.
I1014 19:27:41.672312  2010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.149s (election complete)
I1014 19:27:41.672366  2010 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 FOLLOWER]: Leader election won for term 681
I1014 19:27:41.672444  2010 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:41.672638 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 680 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.672785 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 680. Current term is 681.
I1014 19:27:41.672680  2010 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 572, Committed index: 572, Last appended: 679.572, Last appended by leader: 572, Current term: 681, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
W1014 19:27:41.673120 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 680 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 680. Current term is 681.
I1014 19:27:41.673198 28594 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 680 election: Cancelling election due to peer responding with higher term
I1014 19:27:41.673354 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 681.
I1014 19:27:41.673476  1839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:41.673573  1839 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Leader election lost for term 680. Reason: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 680. Current term is 681.
I1014 19:27:41.673580  2012 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.673753  2012 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.673802 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 681 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.673946  2011 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:41.673954 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 680 candidate_status { last_received { term: 679 index: 572 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.674079 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 680 because replica is either leader or believes a valid leader to be alive.
W1014 19:27:41.674433 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 680 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 680 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.675369  2011 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:41.676297  2011 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:41.677017  2011 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:41.678686 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 679 index: 572. Preceding OpId from leader: term: 681 index: 574. (index mismatch)
I1014 19:27:41.678722 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 679 index: 572. Preceding OpId from leader: term: 681 index: 574. (index mismatch)
I1014 19:27:41.679049  2010 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 573, Last known committed idx: 572, Time since last communication: 0.000s
I1014 19:27:41.679239  2012 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 573, Last known committed idx: 572, Time since last communication: 0.000s
I1014 19:27:41.680737   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.680974   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.680928  1839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.681185  1839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.681840 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.682027 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.682282 32224 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.682480 32224 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.682610   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.682914   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.683274  1839 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 681 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.683571  1839 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.782996   842 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:41.783180   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:41.783252   842 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 681 FOLLOWER]: Advancing to term 682
I1014 19:27:41.785758   842 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.786332   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 682 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:41.786561   842 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 682 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.786976 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 682 candidate_status { last_received { term: 681 index: 574 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:41.787163 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 682 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.787259 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 682 candidate_status { last_received { term: 681 index: 574 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.787451 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 682 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.787807 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 682 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 682 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.788002 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 681. Current term is 682. Ops: []
I1014 19:27:41.788003 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 682 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 682 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:41.788105 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 682 election: Election decided. Result: candidate lost.
I1014 19:27:41.788350   842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.110s (election complete)
I1014 19:27:41.788471   842 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Leader election lost for term 682. Reason: could not achieve majority
I1014 19:27:41.788466  2010 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 681.574, Next index: 575, Last known committed idx: 574, Time since last communication: 0.000s
I1014 19:27:41.788672  2010 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Stepping down as leader of term 681
I1014 19:27:41.788699  2010 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:41.788818  2010 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 574, Committed index: 574, Last appended: 681.574, Last appended by leader: 574, Current term: 681, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.788955  2010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:41.788995  2010 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:41.789106  2010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 681 FOLLOWER]: Advancing to term 682
I1014 19:27:41.868733  1839 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:41.868824  1839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:41.868896  1839 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 681 FOLLOWER]: Advancing to term 682
I1014 19:27:41.876314  1839 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.876751  1839 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 682 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.876902  1839 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 682 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:41.877537 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 682 candidate_status { last_received { term: 681 index: 574 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:41.877609 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 682 candidate_status { last_received { term: 681 index: 574 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:41.877723 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 682: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:41.877734 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.127s (vote granted)
I1014 19:27:41.878168 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 682 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 682: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:41.879288 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 682.
I1014 19:27:41.879611 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 682 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:41.879660 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 682 election: Election decided. Result: candidate won.
I1014 19:27:41.879822  1839 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.139s (election complete)
I1014 19:27:41.879876  1839 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 FOLLOWER]: Leader election won for term 682
I1014 19:27:41.879962  1839 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:41.880159  1839 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 574, Committed index: 574, Last appended: 681.574, Last appended by leader: 574, Current term: 682, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:41.881261  1782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.881472  1782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.882205  2135 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:41.883585  2135 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:41.884531  2135 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:41.885130  2135 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:41.887034 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 681 index: 574. Preceding OpId from leader: term: 682 index: 576. (index mismatch)
I1014 19:27:41.887050 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 681 index: 574. Preceding OpId from leader: term: 682 index: 576. (index mismatch)
I1014 19:27:41.887527  1782 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 575, Last known committed idx: 574, Time since last communication: 0.000s
I1014 19:27:41.887682  1839 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 575, Last known committed idx: 574, Time since last communication: 0.000s
I1014 19:27:41.889201   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.889391   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.889257  2010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.889467  2010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.890828  1782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.891058  1782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.891227  1782 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.891388  1782 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:41.891422   842 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.891626   842 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:41.892546  2010 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 682 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:41.892683  2010 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.407215  2425 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:42.407352  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:27:42.407447  2425 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 682 FOLLOWER]: Advancing to term 683
I1014 19:27:42.409098  2425 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.409600  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 683 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.409934  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 683 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:42.410151 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 683 candidate_status { last_received { term: 682 index: 576 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:42.410320 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 683 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.410727 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 683 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 683 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.411180 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 683 candidate_status { last_received { term: 682 index: 576 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:42.411345 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 683 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.411626 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 683 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 683 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.411692 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 683 election: Election decided. Result: candidate lost.
I1014 19:27:42.411877  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.142s (election complete)
I1014 19:27:42.411939  2425 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Leader election lost for term 683. Reason: could not achieve majority
I1014 19:27:42.416802 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 682. Current term is 683. Ops: []
I1014 19:27:42.417953  2136 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 682.576, Next index: 577, Last known committed idx: 576, Time since last communication: 0.000s
I1014 19:27:42.419023  2433 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 LEADER]: Stepping down as leader of term 682
I1014 19:27:42.419096  2433 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:42.419739  2433 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 576, Committed index: 576, Last appended: 682.576, Last appended by leader: 576, Current term: 682, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.420004  2433 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:42.420078  2433 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:42.420243  2433 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 682 FOLLOWER]: Advancing to term 683
I1014 19:27:42.455926  2434 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:42.455996  2434 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:42.456048  2434 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 682 FOLLOWER]: Advancing to term 683
I1014 19:27:42.457798  2434 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.458248  2434 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 683 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.458575  2434 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 683 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:42.458961 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 683 candidate_status { last_received { term: 682 index: 576 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:42.459117 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:42.459115 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 683 candidate_status { last_received { term: 682 index: 576 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:42.459291 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 683: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:42.459640 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 683 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 683: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:42.460541 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 683.
I1014 19:27:42.460799 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 683 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.460846 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 683 election: Election decided. Result: candidate won.
I1014 19:27:42.460988  2434 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.125s (election complete)
I1014 19:27:42.461040  2434 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 FOLLOWER]: Leader election won for term 683
I1014 19:27:42.461305  2434 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:42.461465  2434 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 576, Committed index: 576, Last appended: 682.576, Last appended by leader: 576, Current term: 683, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.462263  2435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.462523  2435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.462956  2436 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:42.464361  2436 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:42.465282  2436 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:42.465911  2436 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:42.467897 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 682 index: 576. Preceding OpId from leader: term: 683 index: 578. (index mismatch)
I1014 19:27:42.468367 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 682 index: 576. Preceding OpId from leader: term: 683 index: 578. (index mismatch)
I1014 19:27:42.468544  2435 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 577, Last known committed idx: 576, Time since last communication: 0.000s
I1014 19:27:42.468958  2435 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 577, Last known committed idx: 576, Time since last communication: 0.000s
I1014 19:27:42.470427  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.470384  2425 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.470656  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.470659  2425 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.472024  2434 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.472200  2434 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.472100  2435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.472240  2435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.472908  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.473067  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.473424  2425 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 683 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.473577  2425 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.666477  2425 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:42.666568  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.124s (starting election)
I1014 19:27:42.666630  2425 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 683 FOLLOWER]: Advancing to term 684
I1014 19:27:42.668792  2425 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 684 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.669294  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 684 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.669438  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 684 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:42.669896 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 684 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:42.670035 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 684 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.670315 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 684 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:42.670372 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 684 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 684 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.670516 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 684 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.670790 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 684 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 684 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.670845 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 684 election: Election decided. Result: candidate lost.
I1014 19:27:42.671010  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:42.671064  2425 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 684 FOLLOWER]: Leader election lost for term 684. Reason: could not achieve majority
I1014 19:27:42.674973 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 684 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 683. Current term is 684. Ops: []
I1014 19:27:42.675398  2435 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 683.578, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I1014 19:27:42.675638  2435 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 LEADER]: Stepping down as leader of term 683
I1014 19:27:42.675683  2435 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:42.675848  2435 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 683.578, Last appended by leader: 578, Current term: 683, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.676008  2435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:42.676046  2435 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:42.676151  2435 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 683 FOLLOWER]: Advancing to term 684
I1014 19:27:42.752643  2440 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 684 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:42.752857  2440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:42.752929  2440 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 684 FOLLOWER]: Advancing to term 685
I1014 19:27:42.754633  2440 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.755065  2440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 685 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.755214  2440 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 685 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:42.755209  2433 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:42.755268  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.146s (starting election)
I1014 19:27:42.755317  2433 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 683 FOLLOWER]: Advancing to term 684
I1014 19:27:42.755801 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 685 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:42.755803 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 685 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:42.755934 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 684 FOLLOWER]: Advancing to term 685
I1014 19:27:42.756003 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.212s (vote granted)
I1014 19:27:42.757227  2433 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 684 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.757509 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 684 FOLLOWER]: Advancing to term 685
I1014 19:27:42.757567 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.132s (vote granted)
I1014 19:27:42.757674  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 684 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:42.757820  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 684 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:42.757930 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 685.
I1014 19:27:42.758297 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 685 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:42.758349 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 685 election: Election decided. Result: candidate won.
I1014 19:27:42.758358 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 684 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:42.758527 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 684. Current term is 685.
I1014 19:27:42.758647 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 684 candidate_status { last_received { term: 683 index: 578 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:42.758786 29010 raft_consensus.cc:2160] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 684. Current term is 685.
W1014 19:27:42.758939 28600 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 684 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 684. Current term is 685.
I1014 19:27:42.758967 28600 leader_election.cc:374] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 684 election: Cancelling election due to peer responding with higher term
I1014 19:27:42.758997  2440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.101s (election complete)
I1014 19:27:42.759050  2440 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Leader election won for term 685
I1014 19:27:42.759054 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 685.
I1014 19:27:42.759130  2440 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
W1014 19:27:42.759171 28594 leader_election.cc:371] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 684 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 684. Current term is 685.
I1014 19:27:42.759177  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:42.759224  2433 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Leader election lost for term 684. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for earlier term 684. Current term is 685.
I1014 19:27:42.759433 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 685 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.759335  2440 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 683.578, Last appended by leader: 578, Current term: 685, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.760139  2435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.760269  2435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.760427  2436 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:42.761731  2436 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:42.762665  2436 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:42.763252  2436 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:42.764839 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 683 index: 578. Preceding OpId from leader: term: 685 index: 580. (index mismatch)
I1014 19:27:42.765063 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 683 index: 578. Preceding OpId from leader: term: 685 index: 580. (index mismatch)
I1014 19:27:42.765223  2440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I1014 19:27:42.765458  2435 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 579, Last known committed idx: 578, Time since last communication: 0.000s
I1014 19:27:42.766682  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.766866  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.766997  2425 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.767132  2425 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.767685  2435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.767873  2435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.767994  2435 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.768169  2435 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:42.768507  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.769068  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.769179  2425 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 685 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:42.769353  2425 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:42.954887  2425 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:42.954962  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:42.955018  2425 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 685 FOLLOWER]: Advancing to term 686
I1014 19:27:42.956859  2425 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:42.957286  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 686 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:42.957417  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 686 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:42.957837 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 686 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:42.957983 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.958269 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 686 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.958220 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 686 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:42.958387 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.958674 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 686 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:42.958729 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 686 election: Election decided. Result: candidate lost.
I1014 19:27:42.958878  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.116s (election complete)
I1014 19:27:42.958930  2425 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Leader election lost for term 686. Reason: could not achieve majority
I1014 19:27:43.037529  2433 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:43.037626  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:43.037672  2433 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 685 FOLLOWER]: Advancing to term 686
I1014 19:27:43.039885  2433 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 686 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.040396  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 686 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.040578  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 686 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.041157 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 686 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.041224 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 686 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.041347 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.041399 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.041725 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 686 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.041934 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 686 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 686 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.041995 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 686 election: Election decided. Result: candidate lost.
I1014 19:27:43.042045 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 686 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 685. Current term is 686. Ops: []
I1014 19:27:43.042539  2440 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 685.580, Next index: 581, Last known committed idx: 580, Time since last communication: 0.000s
I1014 19:27:43.042760  2557 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Stepping down as leader of term 685
I1014 19:27:43.042789  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:43.042809  2557 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:43.042878  2433 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 686 FOLLOWER]: Leader election lost for term 686. Reason: could not achieve majority
I1014 19:27:43.043004  2557 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 580, Committed index: 580, Last appended: 685.580, Last appended by leader: 580, Current term: 685, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.043323  2557 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:43.043357  2557 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:43.043458  2557 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 685 FOLLOWER]: Advancing to term 686
I1014 19:27:43.075228  2425 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:43.075301  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (starting election)
I1014 19:27:43.075357  2425 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 686 FOLLOWER]: Advancing to term 687
I1014 19:27:43.076795  2425 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.077183  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 687 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.077322  2425 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 687 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.077847 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 687 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.077801 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 687 candidate_status { last_received { term: 685 index: 580 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:43.077991 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 686 FOLLOWER]: Advancing to term 687
I1014 19:27:43.078034 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 686 FOLLOWER]: Advancing to term 687
I1014 19:27:43.078055 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (vote granted)
I1014 19:27:43.078101 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.190s (vote granted)
I1014 19:27:43.080047 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 687.
I1014 19:27:43.080114 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 687 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 687.
I1014 19:27:43.080538 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 687 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.080590 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 687 election: Election decided. Result: candidate won.
I1014 19:27:43.080809 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 687 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.081058  2425 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.219s (election complete)
I1014 19:27:43.081116  2425 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 FOLLOWER]: Leader election won for term 687
I1014 19:27:43.081449  2425 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:43.081619  2425 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 580, Committed index: 580, Last appended: 685.580, Last appended by leader: 580, Current term: 687, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.082379  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.082654  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.083050  2700 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:43.084525  2700 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:43.085497  2700 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:43.086092  2700 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:43.087993 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 685 index: 580. Preceding OpId from leader: term: 687 index: 582. (index mismatch)
I1014 19:27:43.088003 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 687 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 685 index: 580. Preceding OpId from leader: term: 687 index: 582. (index mismatch)
I1014 19:27:43.088507  2425 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 581, Last known committed idx: 580, Time since last communication: 0.000s
I1014 19:27:43.089115  2699 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 581, Last known committed idx: 580, Time since last communication: 0.000s
I1014 19:27:43.089960  2557 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.090210  2557 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.091724  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.091933  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.091086  2425 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.092100  2425 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.092041  2557 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.092198  2557 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.092581  2701 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.092855  2701 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.093521  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 687 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.093669  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.288756  2433 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 687 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:43.288831  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.142s (starting election)
I1014 19:27:43.288882  2433 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 687 FOLLOWER]: Advancing to term 688
I1014 19:27:43.290905  2433 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.291457  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 688 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.291656  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 688 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.292275 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 688 candidate_status { last_received { term: 687 index: 582 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.292484 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 688 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.292374 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 688 candidate_status { last_received { term: 687 index: 582 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.292587 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 688 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.292917 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 688 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 688 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.293227 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 688 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 688 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.293288 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 688 election: Election decided. Result: candidate lost.
I1014 19:27:43.293632  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.129s (election complete)
I1014 19:27:43.293711  2433 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Leader election lost for term 688. Reason: could not achieve majority
I1014 19:27:43.298822 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 687. Current term is 688. Ops: []
I1014 19:27:43.299262  2701 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 687.582, Next index: 583, Last known committed idx: 582, Time since last communication: 0.000s
I1014 19:27:43.299517  2699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 LEADER]: Stepping down as leader of term 687
I1014 19:27:43.299584  2699 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:43.299757  2699 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 582, Committed index: 582, Last appended: 687.582, Last appended by leader: 582, Current term: 687, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.299888  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:43.299926  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:43.300031  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 687 FOLLOWER]: Advancing to term 688
I1014 19:27:43.361078  2557 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:43.361158  2557 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.128s (starting election)
I1014 19:27:43.361217  2557 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 687 FOLLOWER]: Advancing to term 688
I1014 19:27:43.362730  2557 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.363090  2557 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 688 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.363238  2557 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 688 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.363672 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 688 candidate_status { last_received { term: 687 index: 582 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:43.363765 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 688 candidate_status { last_received { term: 687 index: 582 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.363855 28640 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 688: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:43.363914 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (vote granted)
I1014 19:27:43.364796 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 688 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 688: Already voted for candidate 88e69d68a7914bf48d802a2f2e500b71 in this term.
I1014 19:27:43.365332 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 688 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 688.
I1014 19:27:43.365615 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 688 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.365664 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 688 election: Election decided. Result: candidate won.
I1014 19:27:43.365809  2557 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.102s (election complete)
I1014 19:27:43.365864  2557 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 FOLLOWER]: Leader election won for term 688
I1014 19:27:43.365952  2557 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:43.366109  2557 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 582, Committed index: 582, Last appended: 687.582, Last appended by leader: 582, Current term: 688, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.366835  2440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.366981  2440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.367334  2819 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:43.368656  2819 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:43.369547  2819 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:43.370106  2819 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:43.371892 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 687 index: 582. Preceding OpId from leader: term: 688 index: 584. (index mismatch)
I1014 19:27:43.371892 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 688 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 687 index: 582. Preceding OpId from leader: term: 688 index: 584. (index mismatch)
I1014 19:27:43.372697  2440 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 583, Last known committed idx: 582, Time since last communication: 0.000s
I1014 19:27:43.373001  2557 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 583, Last known committed idx: 582, Time since last communication: 0.000s
I1014 19:27:43.374191  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.374369  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.374624  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.374830  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.375582  2440 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.375816  2440 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.376094  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.376255  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.375710  2557 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.376601  2557 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.376981  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 688 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.377163  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.477404  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 688 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:43.477510  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.106s (starting election)
I1014 19:27:43.477560  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 688 FOLLOWER]: Advancing to term 689
I1014 19:27:43.479554  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.480005  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 689 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.480144  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 689 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.480794 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 689 candidate_status { last_received { term: 688 index: 584 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:43.480850 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 689 candidate_status { last_received { term: 688 index: 584 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.480991 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 689 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.481029 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 689 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.481518 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 689 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 689 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.481693 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 689 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 689 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.481750 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 689 election: Election decided. Result: candidate lost.
I1014 19:27:43.481952  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (election complete)
I1014 19:27:43.482015  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Leader election lost for term 689. Reason: could not achieve majority
I1014 19:27:43.493116 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 688. Current term is 689. Ops: []
I1014 19:27:43.493746  2557 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 688.584, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I1014 19:27:43.493994  2820 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 LEADER]: Stepping down as leader of term 688
I1014 19:27:43.494045  2820 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:43.494243  2820 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 584, Committed index: 584, Last appended: 688.584, Last appended by leader: 584, Current term: 688, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.494585  2820 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:43.494626  2820 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:43.494751  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 688 FOLLOWER]: Advancing to term 689
I1014 19:27:43.570140  2433 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:43.570227  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:43.570291  2433 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 688 FOLLOWER]: Advancing to term 689
I1014 19:27:43.572237  2433 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.572698  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 689 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.572834  2433 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 689 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.573232 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 689 candidate_status { last_received { term: 688 index: 584 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.573381 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.126s (vote granted)
I1014 19:27:43.573375 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 689 candidate_status { last_received { term: 688 index: 584 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.573551 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 689: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:43.573848 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 689 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 689: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:43.574787 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 689 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 689.
I1014 19:27:43.575049 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 689 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.575093 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 689 election: Election decided. Result: candidate won.
I1014 19:27:43.575251  2433 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.120s (election complete)
I1014 19:27:43.575314  2433 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 FOLLOWER]: Leader election won for term 689
I1014 19:27:43.575604  2433 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:43.575765  2433 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 584, Committed index: 584, Last appended: 688.584, Last appended by leader: 584, Current term: 689, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.576479  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.576792  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.577251  2877 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:43.578634  2877 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:43.579684  2877 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:43.580274  2877 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:43.582121 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 688 index: 584. Preceding OpId from leader: term: 689 index: 586. (index mismatch)
I1014 19:27:43.582159 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 689 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 688 index: 584. Preceding OpId from leader: term: 689 index: 586. (index mismatch)
I1014 19:27:43.582540  2433 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I1014 19:27:43.582661  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I1014 19:27:43.584203  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.584435  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.584334  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.584580  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.585381  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.585585  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.586014  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.586207  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.587133  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.587347  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.587476  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 689 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.587636  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.666820 29303 heartbeater.cc:467] Master 127.20.201.61:59248 was elected leader, sending a full tablet report...
I1014 19:27:43.686002  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:43.686080  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (starting election)
I1014 19:27:43.686128  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 689 FOLLOWER]: Advancing to term 690
I1014 19:27:43.686561  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 689 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:43.686619  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.121s (starting election)
I1014 19:27:43.686657  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 689 FOLLOWER]: Advancing to term 690
I1014 19:27:43.687608  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 690 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.687939  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 690 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.688057  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 690 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.688609 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 690 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:43.688616 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 690 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.688637  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.688763 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 690 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.688921 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 690: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:43.689051  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 690 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:43.689067 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 690 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 690 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.689182  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 690 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.689236 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 690 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 690: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:43.689296 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 690 election: Election decided. Result: candidate lost.
I1014 19:27:43.689648 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 690 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:43.689803 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 690 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.689783 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 690 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.689937 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 690 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 690: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:43.689980  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:43.690032  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 690 FOLLOWER]: Leader election lost for term 690. Reason: could not achieve majority
I1014 19:27:43.690235 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 690 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 690 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 690: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:43.690505 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 690 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 690 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:43.690577 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 690 election: Election decided. Result: candidate lost.
I1014 19:27:43.690743  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:43.690796  2820 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Leader election lost for term 690. Reason: could not achieve majority
I1014 19:27:43.691254 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 689. Current term is 690. Ops: []
I1014 19:27:43.691679  2876 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 689.586, Next index: 587, Last known committed idx: 586, Time since last communication: 0.000s
I1014 19:27:43.691885  2878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Stepping down as leader of term 689
I1014 19:27:43.691939  2878 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:43.692118  2878 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 586, Committed index: 586, Last appended: 689.586, Last appended by leader: 586, Current term: 689, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.692718  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:43.692760  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:43.692876  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 689 FOLLOWER]: Advancing to term 690
I1014 19:27:43.799218  2878 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 690 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:43.799294  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.138s (starting election)
I1014 19:27:43.799348  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 690 FOLLOWER]: Advancing to term 691
I1014 19:27:43.800922  2878 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.801337  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 691 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.801493  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 691 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.801942 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 691 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:43.802047 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 691 candidate_status { last_received { term: 689 index: 586 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:43.802117 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 690 FOLLOWER]: Advancing to term 691
I1014 19:27:43.802170 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 690 FOLLOWER]: Advancing to term 691
I1014 19:27:43.802181 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.189s (vote granted)
I1014 19:27:43.802232 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.202s (vote granted)
I1014 19:27:43.803531 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 691.
I1014 19:27:43.803570 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 691 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 691.
I1014 19:27:43.803877 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 691 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:43.803975 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 691 election: Election decided. Result: candidate won.
I1014 19:27:43.804078 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 691 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:43.804184  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.145s (election complete)
I1014 19:27:43.804237  2878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 FOLLOWER]: Leader election won for term 691
I1014 19:27:43.804325  2878 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:43.804543  2878 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 586, Committed index: 586, Last appended: 689.586, Last appended by leader: 586, Current term: 691, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:43.805335  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.805564  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.805794  2877 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:43.807122  2877 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:43.808099  2877 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:43.808732  2877 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:43.810328 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 691 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 689 index: 586. Preceding OpId from leader: term: 691 index: 588. (index mismatch)
I1014 19:27:43.810580 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 689 index: 586. Preceding OpId from leader: term: 691 index: 588. (index mismatch)
I1014 19:27:43.810752  2878 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 587, Last known committed idx: 586, Time since last communication: 0.000s
I1014 19:27:43.810964  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 587, Last known committed idx: 586, Time since last communication: 0.000s
I1014 19:27:43.812397  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.812314  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.812615  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.812615  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.813338  2433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.813546  2433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.813496  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.813725  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:43.814309  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.814481  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:43.814333  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 691 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:43.814615  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.160575  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 691 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:44.160655  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.129s (starting election)
I1014 19:27:44.160704  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 691 FOLLOWER]: Advancing to term 692
I1014 19:27:44.162808  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.163278  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 692 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.163465  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 692 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.164291 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 692 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.164351 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 692 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.164471 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 692 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.164538 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 692 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.164844 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 692 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 692 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.164983 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 692 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 692 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.165035 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 692 election: Election decided. Result: candidate lost.
I1014 19:27:44.165223  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.109s (election complete)
I1014 19:27:44.165282  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Leader election lost for term 692. Reason: could not achieve majority
I1014 19:27:44.172336 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 691. Current term is 692. Ops: []
I1014 19:27:44.172853  2876 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 691.588, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.173121  2878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 LEADER]: Stepping down as leader of term 691
I1014 19:27:44.173168  2878 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.173338  2878 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 691.588, Last appended by leader: 588, Current term: 691, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.173549  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:44.173590  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:44.173707  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 691 FOLLOWER]: Advancing to term 692
I1014 19:27:44.264474  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:44.264550  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:44.264592  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 691 FOLLOWER]: Advancing to term 692
I1014 19:27:44.266052  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.266510  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 692 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.266680  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 692 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.267232 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 692 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.267285 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 692 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.267441 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.130s (vote granted)
I1014 19:27:44.267496 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 692: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:44.267911 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 692 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e in current term 692: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:44.268993 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 692.
I1014 19:27:44.269387 28786 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 692 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.269466 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 692 election: Election decided. Result: candidate won.
I1014 19:27:44.269829  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.122s (election complete)
I1014 19:27:44.269898  2820 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 FOLLOWER]: Leader election won for term 692
I1014 19:27:44.270181  2820 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:44.270354  2820 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 691.588, Last appended by leader: 588, Current term: 692, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.271164  3148 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 692 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.271298  3148 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.271672  3149 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:44.272979  3149 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:44.273912  3149 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:44.274554  3149 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:44.275024  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:44.275077  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:44.275123  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 692 FOLLOWER]: Advancing to term 693
I1014 19:27:44.276219 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 691 index: 588. Preceding OpId from leader: term: 692 index: 590. (index mismatch)
I1014 19:27:44.276612  2820 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.276875  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.277142 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 692. Current term is 693. Ops: []
I1014 19:27:44.277309  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 693 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.277482  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 693 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.277602  2820 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.277773  3148 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 LEADER]: Stepping down as leader of term 692
I1014 19:27:44.277840  3148 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:44.278192 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 693 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.278076  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 692 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.278345 28639 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 693 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:44.278353  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.278705 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 693 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.278745 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 693 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Service unavailable: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 693 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:44.278731  3148 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 692.590, Last appended by leader: 590, Current term: 692, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.278905  3148 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:44.278967  3148 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:44.279098  3148 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 692 FOLLOWER]: Advancing to term 693
I1014 19:27:44.280462 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 693 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 693 because replica has last-logged OpId of term: 692 index: 590, which is greater than that of the candidate, which has last-logged OpId of term: 691 index: 588.
I1014 19:27:44.280767 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 693 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 693 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 693 because replica has last-logged OpId of term: 692 index: 590, which is greater than that of the candidate, which has last-logged OpId of term: 691 index: 588.
I1014 19:27:44.280818 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 693 election: Election decided. Result: candidate lost.
I1014 19:27:44.280992  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.156s (election complete)
I1014 19:27:44.281056  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Leader election lost for term 693. Reason: could not achieve majority
I1014 19:27:44.378732  2878 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:44.378834  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:44.378873  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 692 FOLLOWER]: Advancing to term 693
I1014 19:27:44.380452  2878 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.380806  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 693 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.380928  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 693 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.381378 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 693 candidate_status { last_received { term: 692 index: 590 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.381527 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (vote granted)
I1014 19:27:44.381450 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 693 candidate_status { last_received { term: 692 index: 590 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.381651 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 693: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:44.382191 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 693 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 693: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:44.382936 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 693 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 693.
I1014 19:27:44.383246 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 693 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.383292 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 693 election: Election decided. Result: candidate won.
I1014 19:27:44.383463  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:44.383556  2878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 FOLLOWER]: Leader election won for term 693
I1014 19:27:44.383639  2878 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.383839  2878 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 692.590, Last appended by leader: 590, Current term: 693, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.384634  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 693 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.384819  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.437767  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:44.437837  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.199s (starting election)
I1014 19:27:44.437889  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 693 FOLLOWER]: Advancing to term 694
I1014 19:27:44.439417  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 694 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.439818  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 694 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.439961  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 694 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.440461 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 694 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.440511 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 694 candidate_status { last_received { term: 691 index: 588 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.440642 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 693 FOLLOWER]: Advancing to term 694
I1014 19:27:44.440666 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 694 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.441028 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 694 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 694 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.442039 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 694 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 694 because replica has last-logged OpId of term: 692 index: 590, which is greater than that of the candidate, which has last-logged OpId of term: 691 index: 588.
I1014 19:27:44.442317 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 694 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 694 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 694 because replica has last-logged OpId of term: 692 index: 590, which is greater than that of the candidate, which has last-logged OpId of term: 691 index: 588.
I1014 19:27:44.442370 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 694 election: Election decided. Result: candidate lost.
I1014 19:27:44.442538  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.297s (election complete)
I1014 19:27:44.442580  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 694 FOLLOWER]: Leader election lost for term 694. Reason: could not achieve majority
I1014 19:27:44.475831 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 694 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 693. Current term is 694. Ops: []
I1014 19:27:44.476354  2876 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 591, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.476661  2878 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 LEADER]: Stepping down as leader of term 693
I1014 19:27:44.476704  2878 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.476833  2878 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 693.591, Last appended by leader: 591, Current term: 693, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.477027  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:44.477059  2878 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:44.477136  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 693 FOLLOWER]: Advancing to term 694
I1014 19:27:44.532307  3148 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 694 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:44.532402  3148 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.135s (starting election)
I1014 19:27:44.532503  3148 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 694 FOLLOWER]: Advancing to term 695
I1014 19:27:44.533937  3148 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.534335  3148 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 695 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.534514  3148 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 695 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.535133 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 695 candidate_status { last_received { term: 692 index: 590 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.535117 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 695 candidate_status { last_received { term: 692 index: 590 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.535261 28640 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 694 FOLLOWER]: Advancing to term 695
I1014 19:27:44.535269 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 694 FOLLOWER]: Advancing to term 695
I1014 19:27:44.535326 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.233s (vote granted)
I1014 19:27:44.536671 28640 raft_consensus.cc:2202] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 695 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 695 because replica has last-logged OpId of term: 693 index: 591, which is greater than that of the candidate, which has last-logged OpId of term: 692 index: 590.
I1014 19:27:44.536689 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 695 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 695.
I1014 19:27:44.537029 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 695 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.537097 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 695 election: Election decided. Result: candidate won.
I1014 19:27:44.537173 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 695 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 695 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 695 because replica has last-logged OpId of term: 693 index: 591, which is greater than that of the candidate, which has last-logged OpId of term: 692 index: 590.
I1014 19:27:44.537281  3148 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (election complete)
I1014 19:27:44.537333  3148 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 FOLLOWER]: Leader election won for term 695
I1014 19:27:44.537396  3148 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:44.537566  3148 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 692.590, Last appended by leader: 590, Current term: 695, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.538267  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 695 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.538473  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.554652  2878 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 695 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:44.554744  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.122s (starting election)
I1014 19:27:44.554814  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 695 FOLLOWER]: Advancing to term 696
I1014 19:27:44.556280  2878 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.556699  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 696 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.556851  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 696 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.557410 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 696 candidate_status { last_received { term: 693 index: 591 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.557477 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 696 candidate_status { last_received { term: 693 index: 591 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.557626 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 696 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.557626 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 695 FOLLOWER]: Advancing to term 696
I1014 19:27:44.557693 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.122s (vote granted)
I1014 19:27:44.558871 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 696 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 696 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.559250 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 696.
I1014 19:27:44.559558 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 696 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.559600 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 696 election: Election decided. Result: candidate won.
I1014 19:27:44.559761  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.107s (election complete)
I1014 19:27:44.559823  2878 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 FOLLOWER]: Leader election won for term 696
I1014 19:27:44.566046  2878 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.566236  2878 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 693.591, Last appended by leader: 591, Current term: 696, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.567049  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.567174  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.626797 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 695. Current term is 696. Ops: []
I1014 19:27:44.627279  2820 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 0.0, Next index: 591, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.627487  3148 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 LEADER]: Stepping down as leader of term 695
I1014 19:27:44.627530  3148 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:44.627688  3148 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 695.591, Last appended by leader: 591, Current term: 695, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.627820  3148 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:44.627851  3148 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:44.627928  3148 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 695 FOLLOWER]: Advancing to term 696
I1014 19:27:44.654654 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 696 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 695 index: 591. Preceding OpId from leader: term: 696 index: 592. (index mismatch)
I1014 19:27:44.655150  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 592, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.655272  2876 consensus_queue.cc:1104] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer cbc42c7d5db44e00883549e254bdd22e log is divergent from this leader: its last log entry 695.591 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 588
I1014 19:27:44.656028 28825 raft_consensus.cc:1028] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 696 FOLLOWER]: Deduplicated request from leader. Original: 691.588->[692.589-696.592]   Dedup: 692.590->[693.591-696.592]
I1014 19:27:44.656184 28825 pending_rounds.cc:76] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting all transactions after (but not including) 590
I1014 19:27:44.656208 28825 pending_rounds.cc:97] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Aborting uncommitted NO_OP operation due to leader change: 695.591
I1014 19:27:44.656242 28825 raft_consensus.cc:2632] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 696 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader
I1014 19:27:44.656679  3148 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.656819  3148 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.658980  3148 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.659147  3148 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.659332  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.659526  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:44.659572  3149 catalog_manager.cc:960] Failed waiting for node to catch up after master election: Illegal state: Node cbc42c7d5db44e00883549e254bdd22e not leader. Raft Consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.659708  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.659910  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.659910  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.660104  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.660432 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 691 index: 588. Preceding OpId from leader: term: 696 index: 592. (index mismatch)
I1014 19:27:44.660842  3154 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 592, Last known committed idx: 588, Time since last communication: 0.000s
I1014 19:27:44.664441  3159 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.664983  3159 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.665232  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.665408  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.664983  3156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.665669  3156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.665153  3158 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.665894  3158 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.669651 29302 heartbeater.cc:467] Master 127.20.201.62:56252 was elected leader, sending a full tablet report...
I1014 19:27:44.717110  3150 catalog_manager.cc:973] Term changed from 693 to 696 while waiting for master leader catchup. Not loading sys catalog metadata
I1014 19:27:44.717321  3150 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:44.718705  3150 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:44.719647  3150 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:44.720247  3150 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:44.721858  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 696 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.722015  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.825259  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 696 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:44.825346  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.145s (starting election)
I1014 19:27:44.825405  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 696 FOLLOWER]: Advancing to term 697
I1014 19:27:44.827509  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 697 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.828025  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 697 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.828193  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 697 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.828627 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 697 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.828814 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 697 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.828984 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 697 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.829167 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 697 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.829285 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 697 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 697 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.829457 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 697 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 697 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.829553 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 697 election: Election decided. Result: candidate lost.
I1014 19:27:44.829833  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.148s (election complete)
I1014 19:27:44.829892  2820 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 697 FOLLOWER]: Leader election lost for term 697. Reason: could not achieve majority
I1014 19:27:44.830001 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 697 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 696. Current term is 697. Ops: []
I1014 19:27:44.831037  2878 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 696.593, Next index: 594, Last known committed idx: 593, Time since last communication: 0.000s
I1014 19:27:44.831254  2876 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 LEADER]: Stepping down as leader of term 696
I1014 19:27:44.831292  2876 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.831444  2876 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 593, Committed index: 593, Last appended: 696.593, Last appended by leader: 593, Current term: 696, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.831641  2876 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:44.831673  2876 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:44.831775  2876 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 696 FOLLOWER]: Advancing to term 697
I1014 19:27:44.912540  2876 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 697 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:44.912611  2876 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.148s (starting election)
I1014 19:27:44.912667  2876 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 697 FOLLOWER]: Advancing to term 698
I1014 19:27:44.914067  2876 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.914465  2876 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 698 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.914615  2876 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 698 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:44.914988 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 698 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.915109 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 697 FOLLOWER]: Advancing to term 698
I1014 19:27:44.915092 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 698 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:44.915179 28825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.201s (vote granted)
I1014 19:27:44.915216 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 698 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.915524 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 698 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 698 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.916561 28825 raft_consensus.cc:2255] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 698.
I1014 19:27:44.916872 28600 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 698 election: Vote granted by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.916923 28600 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 698 election: Election decided. Result: candidate won.
I1014 19:27:44.917069  2876 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:44.917117  2876 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 FOLLOWER]: Leader election won for term 698
I1014 19:27:44.917188  2876 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:44.917348  2876 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 593, Committed index: 593, Last appended: 696.593, Last appended by leader: 593, Current term: 698, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.918064  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.918192  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.918350  3150 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:44.919700  3150 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:44.920168  3158 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:44.920234  3158 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.134s (starting election)
I1014 19:27:44.920279  3158 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 696 FOLLOWER]: Advancing to term 697
I1014 19:27:44.920902  3150 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:44.921739  3150 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:44.922462  3158 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 697 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:44.923017  3158 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 697 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:44.923162  3158 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 697 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:44.923759 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 697 FOLLOWER]: Advancing to term 698
I1014 19:27:44.923712 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 697 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:44.923890 28825 raft_consensus.cc:2160] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 697. Current term is 698.
I1014 19:27:44.923877 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 697 candidate_status { last_received { term: 696 index: 593 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:44.924036 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 697 because replica is either leader or believes a valid leader to be alive.
W1014 19:27:44.924264 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 697 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 697. Current term is 698.
I1014 19:27:44.924291 28985 leader_election.cc:374] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 697 election: Cancelling election due to peer responding with higher term
I1014 19:27:44.924454 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 696 index: 593. Preceding OpId from leader: term: 698 index: 595. (index mismatch)
W1014 19:27:44.924576 28985 leader_election.cc:371] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 697 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 697 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:44.924932  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 593, Time since last communication: 0.000s
I1014 19:27:44.925786 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 696 index: 593. Preceding OpId from leader: term: 698 index: 595. (index mismatch)
I1014 19:27:44.925927  3158 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.100s (election complete)
I1014 19:27:44.925982  3158 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Leader election lost for term 697. Reason: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for earlier term 697. Current term is 698.
I1014 19:27:44.926244  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 593, Time since last communication: 0.000s
I1014 19:27:44.926877  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.927062  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.927574  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.927758  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.928261  3158 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.928437  3158 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.928766  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.928941  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:44.929049  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.929167  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.929946  3158 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 698 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:44.930085  3158 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:44.954057 28617 maintenance_manager.cc:304] P 88e69d68a7914bf48d802a2f2e500b71: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033336
I1014 19:27:44.955139 28616 tablet.cc:1190] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Flush: entering stage 1 (old memrowset already frozen for inserts)
I1014 19:27:44.955185 28616 compaction.cc:926] Selected 1 rowsets to compact:
I1014 19:27:44.955217 28616 compaction.cc:929] memrowset(current size on disk: ~0 bytes)
I1014 19:27:44.955232 28616 tablet.cc:1192] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Memstore in-memory size: 3541 bytes
I1014 19:27:44.955267 28616 tablet.cc:1481] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6306080624323227648 or (T in {6306080624323227648})}]
I1014 19:27:44.962632 28616 multi_column_writer.cc:98] Opened CFile writers for 3 column(s)
I1014 19:27:44.979316 28616 tablet.cc:1567] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Flush: entering phase 2 (starting to duplicate updates in new rowsets)
I1014 19:27:44.980916 28616 tablet.cc:1622] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Flush Phase 2: carrying over any updates which arrived during Phase 1
I1014 19:27:44.981014 28616 tablet.cc:1624] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6306080624323227648 or (T in {6306080624323227648})}]
I1014 19:27:44.985584 28616 tablet.cc:1672] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Flush successful on 2 rows (1 rowsets, 6180 bytes)
I1014 19:27:44.986201 28616 maintenance_manager.cc:494] P 88e69d68a7914bf48d802a2f2e500b71: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s	user 0.018s	sys 0.007s Metrics: {"cfile_init":1,"data dirs.queue_time_us":947,"data dirs.run_cpu_time_us":162,"data dirs.run_wall_time_us":795,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1066,"lbm_writes_lt_1ms":27,"thread_start_us":420,"threads_started":1}
I1014 19:27:44.987900 28617 maintenance_manager.cc:304] P 88e69d68a7914bf48d802a2f2e500b71: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I1014 19:27:44.988742 28616 maintenance_manager.cc:494] P 88e69d68a7914bf48d802a2f2e500b71: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I1014 19:27:45.028681  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:45.028756  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.118s (starting election)
I1014 19:27:45.028806  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 698 FOLLOWER]: Advancing to term 699
I1014 19:27:45.030161  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.030535  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 699 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.030650  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 699 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.031186 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 699 candidate_status { last_received { term: 698 index: 595 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.031371 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 699 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.031731 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 699 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 699 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.038061 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 699 candidate_status { last_received { term: 698 index: 595 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.038239 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 699 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.038739 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 699 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 699 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.038810 28786 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 699 election: Election decided. Result: candidate lost.
I1014 19:27:45.038992  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.114s (election complete)
I1014 19:27:45.039050  2820 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Leader election lost for term 699. Reason: could not achieve majority
I1014 19:27:45.042707 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 698. Current term is 699. Ops: []
I1014 19:27:45.043339  2878 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 698.595, Next index: 596, Last known committed idx: 595, Time since last communication: 0.000s
I1014 19:27:45.043565  2876 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Stepping down as leader of term 698
I1014 19:27:45.043607  2876 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:45.043778  2876 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 595, Committed index: 595, Last appended: 698.595, Last appended by leader: 595, Current term: 698, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.043956  2876 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:45.043990  2876 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:45.044100  2876 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 698 FOLLOWER]: Advancing to term 699
I1014 19:27:45.045318 28990 maintenance_manager.cc:304] P 4043e04130264237805f012d8e251bd4: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033352
I1014 19:27:45.046648 28989 tablet.cc:1190] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Flush: entering stage 1 (old memrowset already frozen for inserts)
I1014 19:27:45.046697 28989 compaction.cc:926] Selected 1 rowsets to compact:
I1014 19:27:45.046736 28989 compaction.cc:929] memrowset(current size on disk: ~0 bytes)
I1014 19:27:45.046759 28989 tablet.cc:1192] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Memstore in-memory size: 3541 bytes
I1014 19:27:45.046841 28989 tablet.cc:1481] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6306080624323227648 or (T in {6306080624323227648})}]
I1014 19:27:45.053879 28989 multi_column_writer.cc:98] Opened CFile writers for 3 column(s)
I1014 19:27:45.076617 28989 tablet.cc:1567] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Flush: entering phase 2 (starting to duplicate updates in new rowsets)
I1014 19:27:45.076983 28989 tablet.cc:1622] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Flush Phase 2: carrying over any updates which arrived during Phase 1
I1014 19:27:45.077016 28989 tablet.cc:1624] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6306080624323227648 or (T in {6306080624323227648})}]
I1014 19:27:45.080051 28989 tablet.cc:1672] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Flush successful on 2 rows (1 rowsets, 6180 bytes)
I1014 19:27:45.080406 28989 maintenance_manager.cc:494] P 4043e04130264237805f012d8e251bd4: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.034s	user 0.027s	sys 0.002s Metrics: {"cfile_init":1,"data dirs.queue_time_us":333,"data dirs.run_cpu_time_us":171,"data dirs.run_wall_time_us":400,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":27,"thread_start_us":212,"threads_started":1}
I1014 19:27:45.081600 28990 maintenance_manager.cc:304] P 4043e04130264237805f012d8e251bd4: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I1014 19:27:45.082101 28989 maintenance_manager.cc:494] P 4043e04130264237805f012d8e251bd4: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I1014 19:27:45.113487  3158 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:45.113562  3158 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.136s (starting election)
I1014 19:27:45.113624  3158 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 698 FOLLOWER]: Advancing to term 699
I1014 19:27:45.115058  3158 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.115459  3158 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 699 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.115623  3158 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 699 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.116116 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 699 candidate_status { last_received { term: 698 index: 595 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.116143 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 699 candidate_status { last_received { term: 698 index: 595 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.116284 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.141s (vote granted)
I1014 19:27:45.116315 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 699: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:45.116659 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 699 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 699: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:45.118109 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 699 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 699.
I1014 19:27:45.118383 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 699 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.118456 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 699 election: Election decided. Result: candidate won.
I1014 19:27:45.118633  3158 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:45.118691  3158 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 FOLLOWER]: Leader election won for term 699
I1014 19:27:45.118785  3158 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:45.118960  3158 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 595, Committed index: 595, Last appended: 698.595, Last appended by leader: 595, Current term: 699, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.119765  3156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.119892  3156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.120306  3286 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:45.124231  3286 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:45.126397  3286 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:45.127717  3286 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:45.129920 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 698 index: 595. Preceding OpId from leader: term: 699 index: 597. (index mismatch)
I1014 19:27:45.129937 28640 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 699 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 698 index: 595. Preceding OpId from leader: term: 699 index: 597. (index mismatch)
I1014 19:27:45.130357  3158 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 596, Last known committed idx: 595, Time since last communication: 0.000s
I1014 19:27:45.130548  3156 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 596, Last known committed idx: 595, Time since last communication: 0.000s
I1014 19:27:45.132055  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.132074  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.132248  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.132248  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.133554  3156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.133755  3156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.134225  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.134405  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.134460  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.134639  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.135190  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 699 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.135365  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.181736 28805 maintenance_manager.cc:304] P cbc42c7d5db44e00883549e254bdd22e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033392
I1014 19:27:45.182888 28804 tablet.cc:1190] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Flush: entering stage 1 (old memrowset already frozen for inserts)
I1014 19:27:45.182934 28804 compaction.cc:926] Selected 1 rowsets to compact:
I1014 19:27:45.182965 28804 compaction.cc:929] memrowset(current size on disk: ~0 bytes)
I1014 19:27:45.182981 28804 tablet.cc:1192] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Memstore in-memory size: 3541 bytes
I1014 19:27:45.183022 28804 tablet.cc:1481] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6306080625166675968 or (T in {6306080625166675968})}]
I1014 19:27:45.189767 28804 multi_column_writer.cc:98] Opened CFile writers for 3 column(s)
I1014 19:27:45.206948 28804 tablet.cc:1567] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Flush: entering phase 2 (starting to duplicate updates in new rowsets)
I1014 19:27:45.207373 28804 tablet.cc:1622] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Flush Phase 2: carrying over any updates which arrived during Phase 1
I1014 19:27:45.207407 28804 tablet.cc:1624] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6306080625166675968 or (T in {6306080625166675968})}]
I1014 19:27:45.215306 28804 tablet.cc:1672] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Flush successful on 2 rows (1 rowsets, 6180 bytes)
I1014 19:27:45.215942 28804 maintenance_manager.cc:494] P cbc42c7d5db44e00883549e254bdd22e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.034s	user 0.022s	sys 0.003s Metrics: {"cfile_init":1,"data dirs.queue_time_us":332,"data dirs.run_cpu_time_us":163,"data dirs.run_wall_time_us":432,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":929,"lbm_writes_lt_1ms":27,"thread_start_us":178,"threads_started":1}
I1014 19:27:45.217540 28805 maintenance_manager.cc:304] P cbc42c7d5db44e00883549e254bdd22e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I1014 19:27:45.218230 28804 maintenance_manager.cc:494] P cbc42c7d5db44e00883549e254bdd22e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I1014 19:27:45.235919  2876 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 699 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:45.235991  2876 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.131s (starting election)
I1014 19:27:45.236032  2876 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 699 FOLLOWER]: Advancing to term 700
I1014 19:27:45.238030  2876 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.238524  2876 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 700 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.238524 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 699. Current term is 700. Ops: []
I1014 19:27:45.238680  2876 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 700 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.239097 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 700 candidate_status { last_received { term: 699 index: 597 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.239276 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 700 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.239794 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 700 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 700 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.239733  3156 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 699.597, Next index: 598, Last known committed idx: 597, Time since last communication: 0.000s
I1014 19:27:45.239975  2699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 LEADER]: Stepping down as leader of term 699
I1014 19:27:45.240029  2699 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:45.240067 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 700 candidate_status { last_received { term: 699 index: 597 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.240200  2699 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 597, Committed index: 597, Last appended: 699.597, Last appended by leader: 597, Current term: 699, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.240344  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:45.240383  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:45.240499  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 699 FOLLOWER]: Advancing to term 700
I1014 19:27:45.241917 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.111s (vote granted)
I1014 19:27:45.243712 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e69d68a7914bf48d802a2f2e500b71 in term 700.
I1014 19:27:45.244098 28594 leader_election.cc:387] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 700 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.244156 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 700 election: Election decided. Result: candidate won.
I1014 19:27:45.244426  2876 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.115s (election complete)
I1014 19:27:45.244508  2876 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 FOLLOWER]: Leader election won for term 700
I1014 19:27:45.244590  2876 raft_consensus.cc:595] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 LEADER]: Becoming Leader. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:45.244750  2876 consensus_queue.cc:205] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 597, Committed index: 597, Last appended: 699.597, Last appended by leader: 597, Current term: 700, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.245919  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.246091  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.246254  3150 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:45.251000  3150 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:45.253607  3150 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:45.255023  3150 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Loaded TSK: 0
I1014 19:27:45.257267 28825 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 699 FOLLOWER]: Advancing to term 700
I1014 19:27:45.257387 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 699 index: 597. Preceding OpId from leader: term: 700 index: 599. (index mismatch)
I1014 19:27:45.257895  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 598, Last known committed idx: 597, Time since last communication: 0.000s
I1014 19:27:45.259148 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 700 FOLLOWER]: Refusing update from remote peer 88e69d68a7914bf48d802a2f2e500b71: Log matching property violated. Preceding OpId in replica: term: 699 index: 597. Preceding OpId from leader: term: 700 index: 599. (index mismatch)
I1014 19:27:45.259651  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.259711  2876 consensus_queue.cc:963] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 598, Last known committed idx: 597, Time since last communication: 0.000s
I1014 19:27:45.259810  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.260978  2876 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.261160  2876 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.261307  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e69d68a7914bf48d802a2f2e500b71. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.261520  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.262305  2878 sys_catalog.cc:337] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.262428  2878 sys_catalog.cc:340] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.263203  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.263387  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.263517  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 700 leader_uuid: "88e69d68a7914bf48d802a2f2e500b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.263674  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.358322 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29036
I1014 19:27:45.364212  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 700 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:45.364297  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.141s (starting election)
I1014 19:27:45.364356  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 700 FOLLOWER]: Advancing to term 701
I1014 19:27:45.365804  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.366142  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 701 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.366262  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 701 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.366801 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 701 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.366883 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 701 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.367000 28640 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 701 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.367067 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 701 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.367385 28786 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 701 election: Vote denied by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252). Message: Invalid argument: T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 LEADER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 701 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.367460 28780 leader_election.cc:401] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 701 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 701 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.367519 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 701 election: Election decided. Result: candidate lost.
I1014 19:27:45.367846  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (election complete)
I1014 19:27:45.367903  2820 raft_consensus.cc:2497] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Leader election lost for term 701. Reason: could not achieve majority
I1014 19:27:45.369102 29299 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:27:45.380990 28825 raft_consensus.cc:1051] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Rejecting Update request from peer 88e69d68a7914bf48d802a2f2e500b71 for earlier term 700. Current term is 701. Ops: []
I1014 19:27:45.381482  2878 consensus_queue.cc:974] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: INVALID_TERM, Last received: 700.599, Next index: 600, Last known committed idx: 599, Time since last communication: 0.000s
I1014 19:27:45.381712  3154 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 LEADER]: Stepping down as leader of term 700
I1014 19:27:45.381754  3154 raft_consensus.cc:630] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 LEADER]: Becoming Follower/Learner. State: Replica: 88e69d68a7914bf48d802a2f2e500b71, State: Running, Role: LEADER
I1014 19:27:45.381973  3154 consensus_queue.cc:228] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 599, Committed index: 599, Last appended: 700.599, Last appended by leader: 599, Current term: 700, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.382097  3154 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:45.382122  3154 consensus_peers.cc:455] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:45.382205  3154 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 700 FOLLOWER]: Advancing to term 701
I1014 19:27:45.440316 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29036
I1014 19:27:45.447677  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Starting leader election (detected failure of leader 88e69d68a7914bf48d802a2f2e500b71)
I1014 19:27:45.447760  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.114s (starting election)
I1014 19:27:45.447809  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 700 FOLLOWER]: Advancing to term 701
I1014 19:27:45.449839  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.450314  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 701 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.451436  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 701 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.452085 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 701 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.452082 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 701 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.452261 28825 raft_consensus.cc:2185] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 701: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:45.452265 28640 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.125s (vote granted)
I1014 19:27:45.452670 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 701 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 in current term 701: Already voted for candidate cbc42c7d5db44e00883549e254bdd22e in this term.
I1014 19:27:45.454146 28640 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 701 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4043e04130264237805f012d8e251bd4 in term 701.
I1014 19:27:45.454512 28985 leader_election.cc:387] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 701 election: Vote granted by peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.454571 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 701 election: Election decided. Result: candidate won.
W1014 19:27:45.454845 28982 connection.cc:512] server connection from 127.20.201.1:33080 recv error: Network error: failed to read from TLS socket (remote: 127.20.201.1:33080): Connection reset by peer (error 104)
W1014 19:27:45.454841 28781 connection.cc:512] server connection from 127.20.201.1:45525 recv error: Network error: failed to read from TLS socket (remote: 127.20.201.1:45525): Connection reset by peer (error 104)
W1014 19:27:45.454840 28600 connection.cc:512] server connection from 127.20.201.1:59030 recv error: Network error: failed to read from TLS socket (remote: 127.20.201.1:59030): Connection reset by peer (error 104)
I1014 19:27:45.455278  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:45.455358 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 28481
I1014 19:27:45.455379  2699 raft_consensus.cc:2549] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 FOLLOWER]: Leader election won for term 701
I1014 19:27:45.455551  2699 raft_consensus.cc:595] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 LEADER]: Becoming Leader. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:45.455773  2699 consensus_queue.cc:205] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 599, Committed index: 599, Last appended: 700.599, Last appended by leader: 599, Current term: 701, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.456080 28660 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:27:45.456656  3156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 701 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.456849  3156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.457022  3286 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:45.459646  3286 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:45.461380  3286 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:45.462719  3286 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Loaded TSK: 0
I1014 19:27:45.465039 28825 raft_consensus.cc:1083] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Refusing update from remote peer 4043e04130264237805f012d8e251bd4: Log matching property violated. Preceding OpId in replica: term: 700 index: 599. Preceding OpId from leader: term: 701 index: 601. (index mismatch)
I1014 19:27:45.465555  2699 consensus_queue.cc:963] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 600, Last known committed idx: 599, Time since last communication: 0.000s
I1014 19:27:45.467079  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4043e04130264237805f012d8e251bd4. Latest consensus state: current_term: 701 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.467335  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.468297  3156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 701 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.468525  3156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.468962  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 701 leader_uuid: "4043e04130264237805f012d8e251bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.469184  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.654398  3154 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 701 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:45.655390  3154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.119s (starting election)
I1014 19:27:45.655478  3154 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 701 FOLLOWER]: Advancing to term 702
I1014 19:27:45.658149  3154 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 702 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.658670  3154 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 702 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.658836  3154 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 702 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.659008 28640 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 702 FOLLOWER]: Rejecting Update request from peer 4043e04130264237805f012d8e251bd4 for earlier term 701. Current term is 702. Ops: []
I1014 19:27:45.659374 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 702 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.659579 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 702 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.659806 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 702 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.659796  3156 consensus_queue.cc:974] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 }, Status: INVALID_TERM, Last received: 0.0, Next index: 600, Last known committed idx: 599, Time since last communication: 0.000s
I1014 19:27:45.660064 29010 raft_consensus.cc:2216] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 702 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.660140  2699 raft_consensus.cc:2790] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 LEADER]: Stepping down as leader of term 701
I1014 19:27:45.660181  2699 raft_consensus.cc:630] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 LEADER]: Becoming Follower/Learner. State: Replica: 4043e04130264237805f012d8e251bd4, State: Running, Role: LEADER
I1014 19:27:45.660307 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 702 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 702 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.660461 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 702 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 LEADER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 702 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.660552 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 702 election: Election decided. Result: candidate lost.
I1014 19:27:45.660383  2699 consensus_queue.cc:228] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 601, Committed index: 601, Last appended: 701.601, Last appended by leader: 601, Current term: 701, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.660599  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Closing peer: cbc42c7d5db44e00883549e254bdd22e
I1014 19:27:45.660645  2699 consensus_peers.cc:455] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:45.660776  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 701 FOLLOWER]: Advancing to term 702
I1014 19:27:45.660830  3154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.128s (election complete)
I1014 19:27:45.660890  3154 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 702 FOLLOWER]: Leader election lost for term 702. Reason: could not achieve majority

=================================================================
==28481==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 44 byte(s) in 1 object(s) allocated from:
    #0 0x531628 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
    #1 0x3e6669c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (/usr/lib64/libstdc++.so.6+0x3e6669c3c8)

SUMMARY: AddressSanitizer: 44 byte(s) leaked in 1 allocation(s).
I1014 19:27:45.725481  2820 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Starting leader election (detected failure of leader 4043e04130264237805f012d8e251bd4)
I1014 19:27:45.725580  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.117s (starting election)
I1014 19:27:45.725637  2820 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 701 FOLLOWER]: Advancing to term 702
I1014 19:27:45.727682  2820 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.728152  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 702 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.728304  2820 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 702 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.729187 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 702 candidate_status { last_received { term: 701 index: 601 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.729363 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.123s (vote granted)
I1014 19:27:45.731467 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 702 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 702.
I1014 19:27:45.731847 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 702 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.731899 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 702 election: Election decided. Result: candidate won.
I1014 19:27:45.732138  2820 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.123s (election complete)
I1014 19:27:45.732192  2820 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 FOLLOWER]: Leader election won for term 702
I1014 19:27:45.732457  2820 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:45.732699  2820 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 601, Committed index: 601, Last appended: 701.601, Last appended by leader: 601, Current term: 702, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.734133  3727 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 702 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.734369  3727 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.734855  3728 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:45.739455  3728 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:45.741868  3728 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:45.743161  3728 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:45.745555 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 702 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 701 index: 601. Preceding OpId from leader: term: 702 index: 603. (index mismatch)
I1014 19:27:45.746064  2820 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I1014 19:27:45.747570  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 702 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.747771  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.749513  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 702 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.749696  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.749604  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 702 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.749789  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
W1014 19:27:45.846122 28786 leader_election.cc:307] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 702 election: RPC error from VoteRequest() call to peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Timed out: RequestConsensusVote RPC to 127.20.201.62:56252 timed out after 0.117s (SENT)
I1014 19:27:45.849915  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 702 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:45.850014  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.145s (starting election)
I1014 19:27:45.850069  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 702 FOLLOWER]: Advancing to term 703
I1014 19:27:45.851558  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 703 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.852044  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 703 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.852227  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 703 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.852906 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 703 candidate_status { last_received { term: 702 index: 603 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.853106 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 703 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.853487 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 703 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 703 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:45.866621 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 703 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 702. Current term is 703. Ops: []
I1014 19:27:45.867179  2820 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 702.603, Next index: 604, Last known committed idx: 603, Time since last communication: 0.000s
I1014 19:27:45.867431  3729 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 LEADER]: Stepping down as leader of term 702
I1014 19:27:45.867504  3729 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:45.867697  3729 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 603, Committed index: 603, Last appended: 702.603, Last appended by leader: 603, Current term: 702, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.867837  3729 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:45.867862  3729 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:45.867951  3729 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 702 FOLLOWER]: Advancing to term 703

=================================================================
==28481==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 44 byte(s) in 1 object(s) allocated from:
    #0 0x531628 in operator new(unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
    #1 0x3e6669c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (/usr/lib64/libstdc++.so.6+0x3e6669c3c8)

SUMMARY: AddressSanitizer: 44 byte(s) leaked in 1 allocation(s).
I1014 19:27:45.907171  2878 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 702 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:45.911275  2878 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.112s (starting election)
I1014 19:27:45.911387  2878 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 702 FOLLOWER]: Advancing to term 703
I1014 19:27:45.913177 28639 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 702 candidate_status { last_received { term: 701 index: 601 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.913363 28640 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 703 candidate_status { last_received { term: 702 index: 603 } } ignore_live_leader: false dest_uuid: "88e69d68a7914bf48d802a2f2e500b71"
I1014 19:27:45.914501  2878 raft_consensus.cc:479] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 703 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.914965 28638 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 703 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 702. Current term is 703. Ops: []
I1014 19:27:45.915403  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 703 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
I1014 19:27:45.916261 28639 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate cbc42c7d5db44e00883549e254bdd22e for term 702 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:45.916419  2878 leader_election.cc:251] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 703 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.916565 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 703 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:45.916729 28825 raft_consensus.cc:2202] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 703 because replica has last-logged OpId of term: 702 index: 603, which is greater than that of the candidate, which has last-logged OpId of term: 700 index: 599.
I1014 19:27:45.916942 28640 raft_consensus.cc:2230] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 703 because replica is already servicing an update from a current leader or another vote.
I1014 19:27:45.917379 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88e69d68a7914bf48d802a2f2e500b71" candidate_term: 703 candidate_status { last_received { term: 700 index: 599 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.917592 29010 raft_consensus.cc:2185] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 703: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:45.918229 28600 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 703 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 for term 703 because replica has last-logged OpId of term: 702 index: 603, which is greater than that of the candidate, which has last-logged OpId of term: 700 index: 599.
I1014 19:27:45.966222  3729 raft_consensus.cc:457] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 703 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:45.966326  3729 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.143s (starting election)
I1014 19:27:45.966388  3729 raft_consensus.cc:2795] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 703 FOLLOWER]: Advancing to term 704
I1014 19:27:45.967896  3729 raft_consensus.cc:479] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.968387  3729 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 704 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:45.968580  3729 leader_election.cc:251] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 704 election: Requesting vote from peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.969202 29010 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc42c7d5db44e00883549e254bdd22e" candidate_term: 704 candidate_status { last_received { term: 702 index: 603 } } ignore_live_leader: false dest_uuid: "4043e04130264237805f012d8e251bd4"
I1014 19:27:45.969382 29010 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 703 FOLLOWER]: Advancing to term 704
I1014 19:27:45.969480 29010 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.115s (vote granted)
I1014 19:27:45.971562 29010 raft_consensus.cc:2255] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 704 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc42c7d5db44e00883549e254bdd22e in term 704.
I1014 19:27:45.971982 28780 leader_election.cc:387] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 704 election: Vote granted by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788)
I1014 19:27:45.972064 28780 leader_election.cc:278] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 704 election: Election decided. Result: candidate won.
I1014 19:27:45.972285  3729 raft_consensus.cc:2743] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Snoozing failure detection for 0.147s (election complete)
I1014 19:27:45.972350  3729 raft_consensus.cc:2549] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 FOLLOWER]: Leader election won for term 704
I1014 19:27:45.972462  3729 raft_consensus.cc:595] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Becoming Leader. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:45.972653  3729 consensus_queue.cc:205] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 603, Committed index: 603, Last appended: 702.603, Last appended by leader: 603, Current term: 704, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:45.973893  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 704 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.974071  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.974236  3728 catalog_manager.cc:1026] Loading table and tablet metadata into memory...
I1014 19:27:45.976738  3728 catalog_manager.cc:1037] Initializing Kudu internal certificate authority...
I1014 19:27:45.978458  3728 catalog_manager.cc:1046] Loading token signing keys...
I1014 19:27:45.979722  3728 catalog_manager.cc:4073] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e: Loaded TSK: 0
I1014 19:27:45.981881 29010 raft_consensus.cc:1083] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 704 FOLLOWER]: Refusing update from remote peer cbc42c7d5db44e00883549e254bdd22e: Log matching property violated. Preceding OpId in replica: term: 702 index: 603. Preceding OpId from leader: term: 704 index: 605. (index mismatch)
I1014 19:27:45.982333  3729 consensus_queue.cc:963] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 604, Last known committed idx: 603, Time since last communication: 0.000s
I1014 19:27:45.984396  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc42c7d5db44e00883549e254bdd22e. Latest consensus state: current_term: 704 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.984611  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
I1014 19:27:45.985379  2820 sys_catalog.cc:337] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 704 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.985587  2820 sys_catalog.cc:340] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [sys.catalog]: This master's current role is: LEADER
I1014 19:27:45.986178  2699 sys_catalog.cc:337] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 704 leader_uuid: "cbc42c7d5db44e00883549e254bdd22e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } } }
I1014 19:27:45.986335  2699 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [sys.catalog]: This master's current role is: FOLLOWER
W1014 19:27:45.998491 28985 leader_election.cc:307] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 703 election: RPC error from VoteRequest() call to peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Timed out: RequestConsensusVote RPC to 127.20.201.62:56252 timed out after 0.145s (SENT)
I1014 19:27:45.998610 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 703 election: Election decided. Result: candidate lost.
I1014 19:27:45.998844  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.127s (election complete)
I1014 19:27:45.998904  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 704 FOLLOWER]: Leader election lost for term 703. Reason: could not achieve majority
I1014 19:27:46.105382 28594 leader_election.cc:401] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 703 election: Vote denied by peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788). Message: Invalid argument: T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 703 FOLLOWER]: Leader election vote request: Denying vote to candidate 88e69d68a7914bf48d802a2f2e500b71 in current term 703: Already voted for candidate 4043e04130264237805f012d8e251bd4 in this term.
I1014 19:27:46.107795 28594 leader_election.cc:278] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [CANDIDATE]: Term 703 election: Election decided. Result: candidate lost.
W1014 19:27:46.108675 28594 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.201.60:53788, user_credentials={real_user=jenkins}} blocked reactor thread for 190291us
I1014 19:27:46.110038  3154 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71: Snoozing failure detection for 0.190s (election complete)
I1014 19:27:46.110209  3154 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 88e69d68a7914bf48d802a2f2e500b71 [term 703 FOLLOWER]: Leader election lost for term 703. Reason: could not achieve majority
I1014 19:27:46.110675 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 28481
W1014 19:27:46.112448 28786 leader_election.cc:307] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [CANDIDATE]: Term 704 election: RPC error from VoteRequest() call to peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Timed out: RequestConsensusVote RPC to 127.20.201.62:56252 timed out after 0.143s (SENT)
W1014 19:27:46.151190 28786 connection.cc:512] client connection to 127.20.201.62:56252 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107)
W1014 19:27:46.151309 28786 consensus_peers.cc:433] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Couldn't send request to peer 88e69d68a7914bf48d802a2f2e500b71 for tablet 00000000000000000000000000000000. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). Retrying in the next heartbeat period. Already tried 1 times.
I1014 19:27:46.151800 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 28664
I1014 19:27:46.152676 28846 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:27:46.206620  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 704 FOLLOWER]: Starting leader election (detected failure of leader cbc42c7d5db44e00883549e254bdd22e)
I1014 19:27:46.206703  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.130s (starting election)
I1014 19:27:46.206759  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 704 FOLLOWER]: Advancing to term 705
I1014 19:27:46.208225  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 705 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:46.208667  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 705 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:46.208818  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 705 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:27:46.209692 28985 leader_election.cc:307] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 705 election: RPC error from VoteRequest() call to peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Network error: Client connection negotiation failed: client connection to 127.20.201.62:56252: connect: Connection refused (error 111)
I1014 19:27:46.337884  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 705 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:46.337982  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.137s (starting election)
I1014 19:27:46.338040  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 705 FOLLOWER]: Advancing to term 706
I1014 19:27:46.339634  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:46.340157  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 706 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:46.340329  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 706 election: Requesting vote from peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248)
W1014 19:27:46.340788 28985 leader_election.cc:307] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 705 election: RPC error from VoteRequest() call to peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248): Timed out: RequestConsensusVote RPC to 127.20.201.61:59248 timed out after 0.130s (SENT)
I1014 19:27:46.340849 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 705 election: Election decided. Result: candidate lost.
I1014 19:27:46.341024  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.138s (election complete)
I1014 19:27:46.341078  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Leader election lost for term 705. Reason: could not achieve majority
W1014 19:27:46.341183 28985 leader_election.cc:307] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 706 election: RPC error from VoteRequest() call to peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Network error: Client connection negotiation failed: client connection to 127.20.201.62:56252: connect: Connection refused (error 111)
I1014 19:27:46.362088 28825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 705 candidate_status { last_received { term: 704 index: 605 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:46.362289 28825 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 705 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:46.362296 28826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4043e04130264237805f012d8e251bd4" candidate_term: 706 candidate_status { last_received { term: 704 index: 605 } } ignore_live_leader: false dest_uuid: "cbc42c7d5db44e00883549e254bdd22e"
I1014 19:27:46.362426 28826 raft_consensus.cc:2216] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 706 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:46.362609 29010 raft_consensus.cc:1051] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Rejecting Update request from peer cbc42c7d5db44e00883549e254bdd22e for earlier term 704. Current term is 706. Ops: []
I1014 19:27:46.363137  3727 consensus_queue.cc:974] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 }, Status: INVALID_TERM, Last received: 704.605, Next index: 606, Last known committed idx: 605, Time since last communication: 0.000s
I1014 19:27:46.363351  2820 raft_consensus.cc:2790] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Stepping down as leader of term 704
I1014 19:27:46.363389  2820 raft_consensus.cc:630] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Becoming Follower/Learner. State: Replica: cbc42c7d5db44e00883549e254bdd22e, State: Running, Role: LEADER
I1014 19:27:46.363703 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 28664
I1014 19:27:46.363579  2820 consensus_queue.cc:228] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 605, Committed index: 605, Last appended: 704.605, Last appended by leader: 605, Current term: 704, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:46.363684 28985 leader_election.cc:401] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 706 election: Vote denied by peer cbc42c7d5db44e00883549e254bdd22e (127.20.201.61:59248). Message: Invalid argument: T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e [term 704 LEADER]: Leader election vote request: Denying vote to candidate 4043e04130264237805f012d8e251bd4 for term 706 because replica is either leader or believes a valid leader to be alive.
I1014 19:27:46.363744  2820 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 4043e04130264237805f012d8e251bd4 (127.20.201.60:53788): Closing peer: 4043e04130264237805f012d8e251bd4
I1014 19:27:46.363759 28985 leader_election.cc:278] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 706 election: Election decided. Result: candidate lost.
I1014 19:27:46.363778  2820 consensus_peers.cc:455] T 00000000000000000000000000000000 P cbc42c7d5db44e00883549e254bdd22e -> Peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252): Closing peer: 88e69d68a7914bf48d802a2f2e500b71
I1014 19:27:46.363996  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.112s (election complete)
I1014 19:27:46.364053  2699 raft_consensus.cc:2497] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Leader election lost for term 706. Reason: could not achieve majority
I1014 19:27:46.404321 21284 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 28851
I1014 19:27:46.405143 29031 generic_service.cc:169] Checking for leaks (request via RPC)
I1014 19:27:46.592005  2699 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I1014 19:27:46.592095  2699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4: Snoozing failure detection for 0.191s (starting election)
I1014 19:27:46.592135  2699 raft_consensus.cc:2795] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 706 FOLLOWER]: Advancing to term 707
I1014 19:27:46.593719  2699 raft_consensus.cc:479] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [term 707 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e69d68a7914bf48d802a2f2e500b71" member_type: VOTER last_known_addr { host: "127.20.201.62" port: 56252 } } peers { permanent_uuid: "cbc42c7d5db44e00883549e254bdd22e" member_type: VOTER last_known_addr { host: "127.20.201.61" port: 59248 } } peers { permanent_uuid: "4043e04130264237805f012d8e251bd4" member_type: VOTER last_known_addr { host: "127.20.201.60" port: 53788 } }
I1014 19:27:46.594281  2699 leader_election.cc:251] T 00000000000000000000000000000000 P 4043e04130264237805f012d8e251bd4 [CANDIDATE]: Term 707 election: Requesting vote from peer 88e69d68a7914bf48d802a2f2e500b71 (127.20.201.62:56252)
I1014 19:27:46.594355 21284 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 28851
[       OK ] MultiMasterConnectToClusterTest.ConnectToCluster (122957 ms)
[----------] 1 test from MultiMasterConnectToClusterTest (122958 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 3 test cases ran. (137418 ms total)
[  PASSED  ] 6 tests.